Tag: Scale up

Azure, Business Intelligence - BI, Fabric Administrator, Fabric Capacity, Logic Apps, Microsoft Fabric, Power BI

Microsoft Fabric: Resolving Capacity Admin Permission Issues in Automate Capacity Scaling with Azure LogicApps

Resolving Capacity Admin Permission Issues in Automate Capacity Scaling with Azure LogicApps

A while back, I published a blogpost explaining how to use Azure LogicApps to automate scaling Microsoft Fabric F capacities under the PAYG (Pay-As-You-Go) licensing option. Some of my followers reported an issue with their Capacity Admin settings when running the LogicApp solution. The issue was that their capacity admins disappeared after they had run the LogicApps to upscale or downscale the capacity. After some investigation, I found out what the problem was. At the same time, some of my other followers suggested a fix which involved hardcoding the admins into the solution. While this would work in some cases, it is not a practical solution in the long run, as the admin settings may evolve over time. This makes the solution hard to maintain and unreliable. Back then, I suggested using the APIs and an HTTP action in a new LogicApps solution. This blog is the continuation of the previous blog and a quick and easy fix that ensures the automation runs smoothly with minimal to no manual work or maintenance afterwards. I have also published a tutorial video on YouTube explaining the process from the beginning (which was already covered in my previous blog, so I do not explain it here again) which you can watch here:

A Reminder of the Previous Solution

I suggest you read my blog about automating Fabric capacity scaling with Azure LogicApps as it provides a step-by-step guide to implement the solution. But if you have already implemented, or you are just after the fix, jump to the next section. The following image shows how the original solution works:

Automate Scaling Microsoft Fabric F Capacities with Azure LogicApps
Automate Scaling Microsoft Fabric F Capacities with Azure LogicApps

Here is a quick explanation of how it works:

  1. The Trigger runs the workflow automatically every hour.
  2. The Read a Resource, which is an Azure Resource Manager operation, reveals information about a resource that, in our implementation, is a Microsoft Fabric Capacity.
  3. A condition to check the Status of the capacity. If the capacity is Paused (the condition is true), then do nothing. This is needed as this method only works when the capacity is Active.
  4. If the capacity is Active (the condition is false), then check the current time to see if it is between 2pm and 4pm. This is the timeframe for which we want to upscale the capacity.
  5. If the condition is True, then upscale the capacity to F8 using another ARM operation: Create or update a resource.
  6. If the condition is False, then set the capacity SKU to F2.

The solution works fine if you do not have any Capacity Admin settings either on Azure Portal or on Admin Portal on Microsoft Fabric. But in many cases, we indeed have capacity admins. Let’s see what the issue is.

The Problem

The issue arises when we add some capacity administrators; that are wiped after running the above solution in its current implementation. The following image shows the Capacity Admin settings on both portals:

Fabric Capacity Admins on Azure Portal and Fabric Admin Portal
Fabric Capacity Admins on Azure Portal and Fabric Admin Portal

The reason if that the Create or update a resource also updates the properties of the resource with the ones we define in LogicApps. Therefore, if we do not add any capacity admins, we literally empty the existing capacity admins. Let’s run the solution again to understand why this is happening. The following image shows my capacity admins are wiped out after running my LogicApp workflow:

Capacity Admins deleted after running LogicApps
Capacity Admins deleted after running LogicApps

The issue is also demonstrated in the tutorial video on YouTube:

Continue reading “Microsoft Fabric: Resolving Capacity Admin Permission Issues in Automate Capacity Scaling with Azure LogicApps” →
Azure, Fabric Administrator, Fabric Capacity, Microsoft Fabric, Power BI

Microsoft Fabric: Capacity Cost Management Part 3, Pause Capacity During Christmas with Azure Logic Apps

Microsoft Fabric: Capacity Cost Management Part 3, Pause Capacity During Christmas with Azure Logic Apps

In the first blog post of this series, I explained that we can Pause and Resume a Microsoft Fabric capacity from Azure Portal. In the second blog and its accompanying YouTube video, I showed you how to automate the Pause and Resume actions in Azure LogicApps so the capacity starts at 8:00 AM and stops at 4:00 PM. While I have already mentioned in those posts, it is worthwhile to mention again that these methods only make sense for PAYG (Pay-As-You-Go) capacities and NOT the Reservation capacities. While the method works fine, you may need more fine-tuning.

Managing operational costs becomes crucial for businesses leveraging Microsoft Fabric capacities when the holiday season approaches. This presents a unique challenge of maintaining efficiency while reducing unnecessary expenses, especially during Christmas when business operations might slow down or pause entirely.

In this post and video, I will extend the discussions from my previous blog and demonstrate how to optimise your Azure Logic Apps to manage Microsoft Fabric capacity during the Christmas holidays.

Extending the Logic Apps Workflow

Existing Setup Recap

In earlier discussions, we’ve explored using Azure Logic Apps to manage Fabric capacity effectively from 8:00 AM to 4:00 PM on regular business days and pausing operations afterwards. This setup ensures that we’re not incurring costs when the capacity isn’t needed, particularly from 4:00 PM to 8:00 AM the next morning, and throughout the weekends. I encourage you to check out my previous post for more information. This is how the existing solution looks like in Azure LogicApps:

Automating Microsoft Fabric Capacity with Azure LogicApps
Automating Microsoft Fabric Capacity with Azure LogicApps

Incorporating Holiday Schedules

The key to extending this setup for the Christmas period lies in integrating specific holiday schedules into your existing workflows using Workflow Definition Language which is used in Azure Logic Apps and Microsoft Flow. The following expression determines if the current date (in New Zealand Standard Time) falls within the period from December 25th of the current year to January 2nd of the next year:

and(
    greaterOrEquals(
        int(
            formatDateTime(
                convertFromUtc(
                    utcNow(), 
                    'New Zealand Standard Time'
                ), 
                'yyyyMMdd'
            )
        ), 
        int(
   concat(
    formatDateTime(
     utcNow()
     , 'yyyy'
     )
    , '1225'
    )
   ) 
    ), 
    lessOrEquals(
        int(
            formatDateTime(
                convertFromUtc(
                    utcNow(), 
                    'New Zealand Standard Time'
                ), 
                'yyyyMMdd'
            )
        ), 
        int(
   concat(
    add(
     int(
      formatDateTime(
       utcNow()
       , 'yyyy'
       )
      )
     ,1
     )
    , '0102'
    )
   )
  )
)

The following section explains how the expression works.

Continue reading “Microsoft Fabric: Capacity Cost Management Part 3, Pause Capacity During Christmas with Azure Logic Apps” →
Azure, Fabric Administrator, Fabric Capacity, Logic Apps, Microsoft Fabric

Microsoft Fabric: Automating Fabric Capacity Scaling with Azure Logic Apps


UPDATE 16 Sep 2024

I have published a new blog explaining how to resolve the issue of Capacity Admins disappering. Check it out here.

NOTE

This method uses the Create or Update a Resource action. This action uses PUT method which messes up with capacity admin settings on the Fabric Admin portal. This is stated by Kevin and later by Wiroj in the comments section. The REST API to scale or down the Fabric Capacity is as follows:

patch https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.Fabric/capacities/{capacityName}?api-version=2023-11-01

If you run the preceding API using PATCH method, you do not need to have your capacity running. In other words, you can run the API even when the capacity is paused. I will post a new blog explaining this but for now, I hope the information provided here is helpful.

Read more in my comment and Michael’s about this topic.

In a previous post I explained how to manage the capacity costs of a Fabric F capacity (under Pay-As-You-Go pricing model) using Logic Apps to Suspend and Resume it.

A customer who read my previous blog asked me “Can we use a similar method to scale up and down before and after specific workloads?”. This blog post is to answer exactly that.

I want to make some important points clear first and before we dig deeper into the solution:

  • The method described in this post works with Fabric F SKUs under Pay-As-You-Go pricing model.
  • If you have a Power BI Premium capacity, then this method is not valid for your case. But you might be interested in the autoscale option for Power BI Premium capacities.
  • Depending on your current workload, scaling down may not work due to resource unavailability.
  • Depending on your workload, this method may take a while to go through.
  • You need to be either a Capacity Admin or a Fabric Admin to successfully implement this method.
  • This method works based on user authentication, however, you may want to use Service Principal or Manage Identity which require more effort but could be a more desirable method in many scenarios.
  • This post explains a very basic scenario, you’re welcome to scale it to your specific needs.
  • You can consider this post as a continuation of the previous post. So if you are unsure you correctly understand what this blog is trying to explain, then I suggest you read my previous post first where I explain the Logic Apps implementation in more detail.

The Problem

I have an F Fabric capacity and I want to upscale it to an upper tier between the pick-time from 8 AM to 12 PM local time, then downscale it to its original tier.

The Solution

There are many ways to do this including using Azure Resource Manager APIs, Manage Azure Resources in PowerShell, or using Azure Resource Manager connector that can be used on Azure Logic Apps, Power Automate Premium, and Power Apps Premium. This post explores the use of Azure Resource Manager connectors in Azure Logic Apps. With that, let’s begin.

  1. On Azure Portal, search for Logic apps
  2. Select the Logic Apps service
Select Azure Logic Apps on Azure Portal
Select Azure Logic Apps on the Azure Portal
  1. Click the Add button
  2. Pick a Subscription from the list
  3. Pick a Resource Group from the list or create a new one
  4. Enter the Logic App name
  5. Select the Region from the list
  6. Select No if you do not require to Enable log analytics
  7. Select Consumption from the Plan type
  8. Click the Review + create button
Create new Logic Apps service on Azure Portal
Create new Logic Apps service on Azure Portal
  1. Click the Create button
Confirm creating new Logic Apps service
Confirm creating new Logic Apps service
Continue reading “Microsoft Fabric: Automating Fabric Capacity Scaling with Azure Logic Apps” →
Azure, Fabric Administrator, Fabric Capacity, Logic Apps, Microsoft Fabric, Power BI, Power BI Premium Capacity, Power BI Service

Microsoft Fabric: Capacity Cost Management Part 2, Automate Pause/Resume Capacity with Azure Logic Apps

Automate Pause Resume Suspend Fabric Capacity with Azure Logic Apps

In the previous blog post, I explained Microsoft Fabric capacities, shedding light on diverse capacity options and how they influence data projects. We delved into Capacity Units (CUs), pricing nuances, and practical cost control methods, including manually scaling and pausing Fabric capacity. Now, we’re taking the next step in our Microsoft Fabric journey by exploring the possibility of automating the pause and resume process. In this blog post, we’ll unlock the secrets to seamlessly managing your Fabric Capacity with automation that helps us save time and resources while optimising the usage of data and analytics workloads.

Right off the bat, this is a rather long blog, so I added a bonus section at the end for those who are reading from the beginning to the end. With that, let’s dive in!

The Problem

As we have learned in the previous blog post, one way to manage our Fabric capacity costs is to pause the capacity while not in use and resume it again when needed. While this can help with cost management, as it is a manual process, it is prone to human error, which makes it impractical in the long run.

The Solution

A more practical solution is to automate a daily process to pause and resume our Fabric capacity automatically. This can be done by running Azure Management APIs. Depending on our expertise, there are several ways to achieve the goal, such as running APIs on running the APIs via PowerShell (scheduling the runs separately), running the APIs via CloudShell, creating a flow in Power Automate, or creating the workflow in Azure Logic Apps. I prefer the latter, so this blog post explains the method.

I also explain the same scenario on my YouTube channel. Here is the video:

Automating Pause and Resume Fabric Capacity with Azure Logic Apps

Here is the scenario: we are going to create an Azure Logic Apps workflow that automatically does the following:

  • Check the time of the day
  • If it is between 8 am to 4 pm:
  • Check the status of the Fabric capacity
  • If the capacity is paused, then resume it, otherwise do nothing
  • If it is after 4 pm and before 8 am:
  • Check the status of the Fabric capacity
  • If the capacity is resumed, then pause it, otherwise do nothing

Follow these steps to implement the scenario in Azure Logic Apps:

  1. Login to Azure Portal and search for “Logic App“
  2. Click the Logic App service
Finding Logic Apps on Azure Portal

This navigates us to the Logic App service. If you currently have existing Logic Apps workflows, they will appear here.

Continue reading “Microsoft Fabric: Capacity Cost Management Part 2, Automate Pause/Resume Capacity with Azure Logic Apps” →