PowerBI Incremental refresh using Python or R

In this blog, I will show how to leverage Python (or R) to implement an incremental refresh in PowerBI using PowerQuery and Python, nothing is really new ( I am sure Imke and Maxim has blogged about it before).

in a previous blog, I showed how to use R & Python integration to load data to a Database

This approach make sense only when you do a lot of heavy transformation and your data source change based on time.

As an example, in my previous job, we receive a new excel file every Monday (300K rows), this file gets approved and corrected every Thursday.

the workflow was:

save the files in a folder, do the transformation, which was fine , but after the first year, it was around 52 files, and although technically you need only to do transformation for the last file, and as PowerBI does not support incremental refresh, twice a week we redo everything, after two years, the refresh took nearly 30 Minutes and sometimes we get out of memory errors.

in the big picture,Half an hour was not that bad (we have a desktop just for refresh), the worst was, you refresh the model and once you finish, you get a new revision and you must refresh again.

Now using Python/R script, the idea is every file get transformed only 1 time, regardless of how many times you refresh, just by exporting the results of the transformation of every file as a csv in a staging folder.  

  • The first run is slow, as it will process all the existing files in Source Data, but the subsequent run, will transform only new files.
  • Let’s say File 2 was revised, all you need to do,is to delete File2.csv and it will be transformed again, but only that file.
  • Ok, if you see step 4, the files are reloaded each time, I am not too much worried about that, as the batch loading of csv files from a folder using PowerQuery is relatively fast (yes, a bit slow compared to R), the bottleneck is rather the transformation.

the code for python script is here, as you can see PowerQuery integration is amazing, just add a new step and you get a dataframe, that’s all,

# 'dataset' holds the input data for this script

df_by_filename = dataset.groupby("filename")

for (filename, filename_df) in df_by_filename:

    filename = filename.replace("zip", "csv")

    filename = filename.replace("PUBLIC_DAILY", "UNIT_PUBLIC_DAILY")    filename_df.to_csv("C:/results/"+filename,index=False)

the script split the dataframe by the column filename, and then export each file separately, currently it is saving into a local folder, but you can easily save those files into a cloud storage

to test it, I built a quick workflow using public data, PBIX here,  the source data is zip files in a public website, there is a new zip file daily, it is relatively complex transformation as you need to unzip the file split it, delete some columns etc, the first run is slow, as it is processing all the files (62 files), but the next run, will just process 1 file, you can simulate that just by deleting some csv files in the staging folder, when you refresh again, only the files deleted will be processed again.

I think the main take away is, Python and R integration are amazing tools to implement new possibilities that will not be necessary available in PowerBI, and you don’t need to be a programmer to use those integration, a serious search on stackoverflow will get you started quickly.

Construction Progress Report – PowerBI – by Darrin Kinney

A quick and easy construction progress and schedule dashboard.

I have previously outlined an approach that can be used for Engineering Progress.

This post is an extension to that which instead of looking at engineering model development, instead looks at construction development. I don’t want to delve too much into the details about exactly how this was built (again see the post above).

Some big differences is that I have used a resource assignment view. in addition to the date metrics This allows for resources histogram and progress curves to be quickly sorted down to an activity level. This approach also follows a prior post Resource Analysis Dashboard .

Construction02

The data

Construction01

The underlying data is very similar to our engineering progress example. We can use a flat file export direct from P6 with a standard set of columns. As I have mentioned before, you can achieve this in a SQL query as part of a larger data model, although with everything, a delicate balance is needed (balancing database formalism and easy excel solution)

We will also have the resource assignment data

Construction06data.JPG

The WBS Slicer and Area Selection

Construction03_wbs

This design element doesn’t work for project with too many WBS elements. For this example, each major area only has about 10 WBS elements, therefore I could pull this off with no drama. I really prefer this selection as opposed to drop downs where it is often difficult to quickly make  selection.

The Pie and Metrics

Construction04pies

Here we follow much of the look and feel I used with the engineering progress; however instead of just using activity count metrics, I have also inserted hour and percent complete metrics. There is nothing fancy about these.

The Data Table

Construction05table.JPG

I’ll sound like a broken record again, when you have a good design with one aspect of a project, you can likely take that and run with it for many other areas. In a following post I will detail this systems engineering aspect to nearly everything we touch.

Obviously the key inclusion into the table is the budget units and %’s. I still prefer these tables views vs the GANTT views. Having clear visibility into the last month dates, the prior month dates,  and variances is the purpose of this view.

The Future

Again, the extension of this are endless. At this stage, we are starting to see how pre filtered views provide more focused dashboard as compared to a one size fits all. Sitting in an EPCM world, most of the detailed activities and schedules are managed by our contractors. Thus, this construction view is more suited to using an export from a contractor Level 4 schedule.

At some point, we will need to begin to discuss an overarching design where a user can navigate to our various dashboard in a logic way.

Happy data wrangling!

Engineering Progress Report – PowerBI – by Darrin Kinney

In this article, I will run through all the steps required to produce an elegant Engineering Progress Report.

Eng12

The intent is not to delve into the manner in which the progress or schedule are updated. I have assumed you have a schedule and progress status for each key area. It is quite amazing how easy is to generate this dashboard, and also the extensions available to use this not just for engineering, but for fabrication, material deliveries, major milestones, contractor key activities, etc.

I will outline the format for our 2 key datasets and then follow with the creation of 2 dashboards: An Overall Status Gauge, and the full detail EPR Dashboard

P6 Schedule Data

Below is our data set we want to use. This data set has been specifically tailored to our resulting visual. Thus, instead of linking directly to an XER, importing into a data model, and performing perhaps too much data work, a nice trick is to instead define specific VIEWS inside P6, so that you can easily copy-paste directly into Excel, then import directly into your dashboard. Thus, the below can be quickly generated each schedule update cycle.

Eng_9

A very nice aspect of this data set is the field “TYPE”. It is good practice to tag activities of a specific type (this ties into my belief about using a framework approach to approach controls). Thus, in theory, you can export the entire schedule, and drive many different dashboards by just filtering on different TYPE fields. In this example I have used

  • M090 = 90% Model Review
  • M100 = 100% AFC

Although, consider tagging every concrete pour Activity in your schedule with C010. You can then use that code to drive a similar dashboard for concrete pours: Or you use F100 for Module Fabrication, where we tag the completion activity for each module for use in dashboard. Ultimately you create a catalog of TYPE codes and can go dashboard crazy with how easy this turns out to me.

This data does not have all the fields we will need in our dashboard. Specifically we will want to create a several measures that will allow for a few metrics. We will need to know if an activity is “FINISHED”, “NOT FINISHED”, and “Critically LATE”. Because these fields are dependent on your target audience, its best to leave the generation of these to code (because everyone can code right!). If you wanted to display metrics on “Started”, then your source data would need to include the start date and perhaps the activity status field from P6. Again, its important to understand the relationship between your visual and your data. In this example, I am treating these activities as effectively milestones in which case the concept of “started” doesn’t apply. key conceptual discussions such as this are vital.

Progress Data

The progress data in this example is only overall progress. The intents is to just show an overview for the entire project and quick metrics for model reviews. Ultimately, you would want a “WBS Specific” dashboard that would display more information over the entire lifecycle of that WBS. In that view, you could present the engineering curve and perhaps EVMS metrics.

Strategy – Do not do everything in one place- keep focus

Too often, I see users pushing design features into dashboards, for what appears to just be whimsical value. Dashboards are not meant to answer 100 questions. Its easier to have 100 dashboards each displaying a key metric, as opposed to 1 dashboard displaying 100 metrics. Keep your approach CLEAN and FOCUSED.

Ideally, our progress data will include fields such as Area, WBS. In this example I have pulled data with just 1 data date and only 1 dataseries (Engineering_Overall). Your backend progress data will likely have data from multiple cut off dates and for multiple series.

Our progress data will look like this. The full data set will also contain a series for “Construction_Overall” too. This will be used on our summary page to outline the power in using this approach to progress data.

Eng_02

Linking our Data into PowerBI

In this example, both data files are simply Excel based files with the data converted to table. This allows for the easiest importing (and also allows for quick refresh of data). Housing the data in the excel files can also facilitate a movement to a more digital way of thinking (more on that in another article)

VISUALIZATION 1 – SUMMARY GAUGE

I am a firm believer in Overall Project Flash reports. So, when we think about dashboards we should have a starting point our overall project status. Thus, the elements presented here are only a key subset of metrics and visuals I would expect on a Project Status Report dashboard.

In this example, looking at engineering progress, we want to see what Percent % Complete we are and how that compares against our Planned % Complete.

A Gauge is a good way to provide a quick visual (Bullet charts are other, and really, the skies the limit)

Eng_Gauge

To generate this we need to create 2 measures: Actual % and Planned %. This is where you really need to understand how dashboards work and how databases work. If you feed a computer a data source, it is no innate way of know something as simple as “What is the current %”. Therefore, we need to write some code.

Because of the format of our progress data, we can search for the maximum data date, then find the value of our actual % field on that date. We can follow an identical approach for the Planned %. Depending on your data, you would need to custom build these measures.

Code to generate our measure for Current %

M_Progress_Actual = CALCULATE (
SUM ( data1[Actual] ),
FILTER (
data1,
data1[DataDate] = MAX ( data1[DataDate] ) && data1[Date]=MAX(data1[DataDate]
)
))

Code to generate our measure for Planned % (similarly we could also pull in our Plan late)

M_Progress_Plan = CALCULATE (
SUM ( data1[BL_Early] ),
FILTER (
data1,
data1[DataDate] = MAX ( data1[DataDate] ) && data1[Date]=MAX(data1[DataDate]
)
))

The required fields for the gauge are obviously these 2 measures.

  • Value = M_Progress_Actual
  • Target Value = M_Progress_Plan

We will also need to provide a filter where Series=”Engineering_Overall” (note that this gauge can now be easily reproduced to showcase planned vs actual for all Series inside our data source. Obviously in the image above you can see I created 2 gauges each with a filter for the specific data series. Ultimately if your back end data has multiple data series for progress sliced and diced different ways, all you have to do is adjust your filter and you can display an endless series of graphs. Of, you can fancy with smart slicers too.

VISUALIZATION 2 – Engineering Progress Report

This is perhaps the most easy to read, interactive and intuitive view into engineering I have ever seen. We can immediately filter into what areas are complete, what areas are critical, scroll to see upcoming deliverables and see an overall graph.

Eng12

It might seem we have a lot going on here, but again, this is all driven off 2 quite simple data sources, and for this page, mostly everything here is from 1 schedule driven table.

The Data Table

The Table is just pulling from our Schedule data (although I have inserted a page level filter to only include activities with the TYPE = M100 and M090). Our fields are as

Eng_4

In the above image, you can see I have had to insert a few measures. I don’t want to go into them all. I’ve inserted some conditional formatting into the Actual/Forecast date column. To achieve this, I created a measure Activity_Status_Num

Activity_Status_Num = IF(ISBLANK(Schedule[Float]),1,IF(Schedule[Float]<1,2, 0))

Then, with these values I can select a formatting specific just for that column in the table. This is very nice feature of the tables in PowerBI that can add nice level of polish.

The Donut Charts

Eng_6

A nice feature of the donut chart is the count metric in the middle. It is generated from a nice little bit of code as seen below. We have 2 Donut Charts. One for our 90% activity and another for the 100%. Thus, all we need to do is place a visual level filter on each.

IsFinished = IF(ISBLANK(Schedule[Float]),1,0)
DonutCounts = SUM(Schedule[IsFinished]) &”/”& COUNT(Schedule[ActivityDesc])
In the above, there are 2 measures: “IsFinished” and “DonutCount”. Again if you want anything to display on a dashboard in a digital world, you are going to have to see this type of code

The real power of the Donut Chart is to allow for very quick sorting – after all we want to see the critical late activities right! Just click the red 90% or 100% section.

Eng13

 

Progress Graph

Eng_7

We have a progress graph too. This is effectively a dumb page level graph. It is not linked to a specific progress series for each WBS. So it will not auto update, and our data model does not link these tables. Although, the graph should add context to the overall page. Deviations from the plan curves, should be viewed by a growing number of critically late packages.

Care needs to be made whenever we look at schedule dates and progress graphs. We do not typically create progress graphs at an activity level (although, you can certainty consider it – I would offer caution against going down that route).

EXTENSIONS

This example has show the power simple data sets can have to improve visibility into our projects. This only showcased a few engineering based activities. However, if you read between the lines, you will understand there is nothing “engineering specific” about what I have done. This approach is completely universal. Given this example also included a progress data set for Construction, obviously, the easiest extension will be to link in a few construction activities in the same way.

Planning and Project Management – What is Missing

Can you imagine using Twitter in the world of Construction – I can! Read on..

Recently, it has hit me, just how poor our project management world is. The ability to clearly communicate and understand what is occurring on a project appears completely lost. Planners have absolutely amazing schedules (I do believe this), cost engineers have incredible detail about the cost build ups sliced and diced every which way, our document management systems are full with every manner of communication – BUT we are still left sitting in meetings where everyone is confused with various key information either not available, or buried inside someone own excel file, bounced off 100 emails threads, or a million other permutations.

Issue – Visibility into our Projects Sucks

Why has this happened? In my view, our leadership teams have failed at pushing good practices of project management. Leadership teams have transitioned from “manage the work” to “manage the people” (thank you Edin M for this quote).

Solution – Get Back to Basics

OpenProj_01

Project Management is task management (you can argue, but even all the new people management issues – are still tasks and can be managed in the same way).

We are building something – lets focus on the activities required to build (and engineer, and contract and procure). All the activities that exist in our Primavera schedules! We need these activities front in center and OWNED by project management. Honestly, how can we even have a management discussion and NOT have the schedule front and center. The schedule says what we should be working on, the schedule says when we should finish something, the schedule says what comes next!

I believe that schedules have been forgotten in project management because they are too unwieldy, to abstract, can only be run by P6 jockeys and not the project at large. We need to get our schedules into the hands of those that actually manage the scope.

In the past, our leadership were more in-tuned to schedules and this synergy was easier. However i fear in today’s world, our leadership have lost the tools and dealing with our schedules (no thanks to our reliance on antiquated tools like P6 that perpetuate the need for designated planning teams to operate the software in pure isolation to the real PM teams).

Issue – We need better Project Management tools.

Answer – They exist everywhere!

Commercially built, off the shelf project management software has risen to be one of the dominant fields of software development. The construction world needs to embrace the tools and get back to basics. It is odd in that 20 years ago, the leaders in project management was the construction world. However, when the technology world sprung up, they didn’t have the knowledge we did, so they built their own tools and approaches. I now believe the tide has turned to the point construction project management now severely trails the rest of the business world.

So How Does This Work?

First, the corporate strategy teams need to decide on a platform (hint – USE JIRA).

JIRA Quick Demo

Here is an example of a JIRA typical managemetn page using JIRA.

JIRA_01

This is a short quick example of some substation work. I have populated JIRA with a few activities that might mirror what you currently manage inside your existing schedule. The difference here is that these activities are not updated by a planner, they are activity managed.

This process to push the ownership of schedule tasks to those that actually manage and deliver the scope is where immense value can be gained. Additionally, each activity allows for commentary and discussion. The ability to insert comments and discuss an activity or its relationships with other activities also brings teams together an allows for a focus on touch points to be activity managed.

OpenProject.org Example

However, really the choice of software isn’t that critical, its the work processes you are going to change – the new online PM tools are structurally all the same. What we are pushing is simply “clarity in what we are doing”. We are pushing the management oversight of what we do, into the hands of those that actually manage the scope. Don’t hide your schedule, don’t hide weekly and daily reports inside your document control system – embed everything into what is effectively a social media platform.

In this example, I am using OpenProject.org , however, keep in mind there are a lot of systems that all work similar.

Add activities

The starting point for me would be to add you P6 activities to your tool. This is the natural place to begin. You schedule already has a structure and usually a very good balance of level of detail.

OpenProj_02

In the above, I have added a typical task that will exist in our P6 schedules. Immediately off the bat, we can see we are operating in a distributed web based environment. We have a nice detailed description for this activity and we have the ability to assign this task to a person.

Up to this point, we are a little overlapped with P6. However, what is lacking in P6 is the ability to really discuss and communicate and UPDATE information associated with a task. The ability to pull the task into a proper project management discussion.

OpenProj_04

The above example says more than you can find in any weekly or monthly report. A picture tells a 1000 stories! The picture is also properly assigned to the activity it represents. The activity has a clearly visible Finish date than can be live edited 24/7.

Our new tools are not meant to replace P6. They are meant to force our discussions into properly structured slices of the project. They are meant to clearly communicate the status of activities. They are meant to get everyone onto the same playing fields when discussing something so that 5 different people do not end up with 10 different dates.

An activity only has 1 start and finish date, an activity only has one percent complete. It is maddening when a project manager asks me to insert the contractual dates into a report. Honestly, when you are building something, the contract date is useless in helping you decide “when will this finish”. It was only the starting point. When you get people out of their office view, in into “I need to manage this scope” you quickly understand that the contract date, or even contractors weekly reports are useless. You have to make a determination of when an activity will finish based on what you know at that time – and be proactive in actively editing the dates when required

Empower people to update activities!

OpenProj_05

This is So Simple?

I sit and look at this capability of something I built in 30 minutes on a Sunday morning and really wonder why our Project Management is leading us down what may not be avenues of real improvement to projects. Does our Project Leadership have the vision to accept such simple solutions to improve our communication?

Digital Transformation?

I have discussed this before, digital transformation is all about Keywords – not project management. Real digital transformation is about altering the way we work – not building a dashboard or a database. This is why digital transformation is not working. Provide tools and process to manage the work, enable your staff to manage their own scope, and clearly communicate and update their tasks.

Really think about how you manage your scope and how implementing a more social platform to break down the walls of communication. Understand how this is disruptive to our old ways of working (not updating schedules). Get people talking off just one play-sheet!

Twitter in Construction

I’d like to end this with what I thought was the most amazing application of this new management approach.

Mersey Gateway Twitter Site

I kid you not. During construction, these guys posted nearly daily pictures and updates.

While working on this project from the home office, I got better updates from the project twitter site, then I did from the project manager. Yes, are you finally able to see that solutions exist, creative solutions exist, that can bring construction in the new digital world!

OpenProj_06

 

P6 Date Formats – Quick Data Hacks!

In dealing with P6 data, sometimes what you expect, is not what you get. When it comes to date formats, this is quite relevant. Here is a guide to transform XER and P6 copy-pasted values into proper date formats.

In dealing with P6 data, sometimes what you expect, is not what you get. When it comes to date formats, this is quite relevant. Here is a guide to transform XER and P6 copy-pasted values into proper date formats.

Problems with XER file format

When dealing with a native XER file, you need to be careful because you can’t always use the field you want. Below is a screen print from a typical TASK dataset.

XER_date_format

When an actual start/finish date has been captured (as seen in the “act_start_date” and “act_end_date” fields above), then the dates that are stored in the “early_start_date” and “early_end_date” fields are no longer valid.

Thus, you need a routine to check if an actual start date exists, and use that in lieu of the early dates. Ultimately there is no good way to deal with, excel to write a routine somewhere in your data import routines – if importing in a database. Another option is to edit your XER import excel file to add 2 new columns for simply “start” and “finish” that will run your check for you in the native excel file before you import the data into a database.  This is an easy hack anywhere you manage your data: However, a core issue here is:

Where will you clean your data?

For me excel is often easier, but that does mean your processes will not be fully automated. Either way, the fix is fundamentally as seen below

 START=IF(ISBLANK(act_start_date),early_start_date,act_start_date)

FINISH=IF(ISBLANK(act_end_date),early_end_date,act_end_date)

 

Problems with P6 format (copy-paste)

If you are not dealing with an XER, you will likely simply copy-paste directly from P6 into Excel. Here again, we have to deal with a few (minor annoying) complications. Everyone I know in the planning world deals with this and has their own routines. I wish I could post them all, because some truly elegant solutions exist. The below is not meant to be “do it this way”, just more of an indication that if perhaps you are running into difficulty “this will work!”.

Start Finish
22-Oct-18 A 9-Sep-19
09-Sep-19*
09-Sep-19*
10-Sep-19 10-Sep-19
05-Jul-19* 10-Sep-19
27-May-19 A 10-Sep-19
03-Dec-18 A 10-Sep-19

Above we can see a typical copy-paste result from P6. Obviously, this is filled with non-date formatted cells. This is caused by the ” A” indicator for activities that have an actual date, and a “*” for activities that have some sort of constraint applied. I have seen a few equation based solutions to strip the bad characters out. I find a code based solution to be slightly more elegant. It also means I do not have to deal with adding extra columns to my file, or performing any copy-paste values. But again, I am sure we all have nice solutions.

The easy fix I use: Scrubb_Data().

Sub Scrubb_Data()

Dim xChars As String
xChars = “*”

Sheets(“P6_CURRENT”).Select
ActiveWindow.SmallScroll ToRight:=3
Columns(“V:V”).Select
Selection.TextToColumns Destination:=Range(“V1”), DataType:=xlDelimited, _
TextQualifier:=xlDoubleQuote, ConsecutiveDelimiter:=True, Tab:=False, _
Semicolon:=False, Comma:=False, Space:=True, Other:=True, OtherChar:=xChars, FieldInfo _
:=Array(Array(1, 3), Array(2, 9), Array(3, 9)), TrailingMinusNumbers:=True

End Sub

This routine only works on 1 column at a time. For me, I can simply copy and paste the select statement and insert a routine for each excel column with a P6 date  (typically just start and finish). After pasting in P6 data, I will open the routine and click “play”. You will need a macro enabled file and will need to be at least a little comfortable with “view code”. There are a few options to auto call this function using a button, call the function when you close the file, many different options.. They all require a bit of VBA knowledge, although, like seriously, who isn’t in our world.

For the above, I ripped most of this from a routine someone else wrote (a routine that stripped the ” A” off the string. I had to add the xChars = “*” aspect.

For a find/replace statement, it is possible to use the below string to strip the “*” off. Here we have to use ~*~ because if we use *, it will replace the entire string. Again, a million ways to handle this.

Capture

Conclusion

Again, there are a 100 ways to skin this cat, and all achieve the same result. When we deal with data from P6, I find it so amazing that when you ask 10 people how we deal with the data, you will end up with at least 20 replies. Ultimately,  I believe this issue is a telling critical flaw in the underlying software. In the digital world, we need a completely different paradigm shift in the way we store and manage data.

Specifically in the Project Management world, I doubt anyone who uses tools such as JIRA or DevOps have to deal with annoyances such as this. Thus, perhaps we too should be using those tools!

Digital Strategy (the beginnings) – by Darrin Kinney

Project Controls is the heart of every major construction project. Our teams are the ones that typically wade through information that never stops and constantly changes. Although at the core, the fundamentals of our strategy are likely the same as what everyone who manages data will encounter.

Digital Strategy for project controls is a tricky beast. We are bombarded by systems that all promise nirvana (and many can deliver nirvana for some projects). However, at the core, before we even discuss systems, we need to understand what our strategy is, and what key concepts we want to push. Below are some ideals of mine. With everything, this is a straw-man discussion. Every project will have its own culture, its own focus, its own needs. Your strategic approach has to cater to what you want from your project. Not to mention that the major companies all have quite rigorous project control plans and detailed procedures. These are great documents that do spell out our scope of services.

Project Controls is the heart of every major construction project. Our teams are the ones that typically wade through information that never stops and constantly changes. Although at the core, the fundamentals of our strategy are likely the same as what everyone who manages data will encounter.

The issues are not related to our knowledge of project controls (and project management) fundamentals. We know how to build schedule, track progress, build baselines, budgets, and track costs parsed into 1000s of potentials contracts and POs. We know how to calculate and track Earned Value, CPI, and SPI. We know change management, trends, change orders, etc. We know how to compile all the information into weekly and monthly reporting. We have the procedures and management plans.

We are the technical experts of our discipline.

However, too often, we are not able to step back and understand that the way in which we work can change, needs to change – and more importantly – the technology that is available now can be utilized to seriously disrupt the way in which we apply our technical knowledge and skills.

Below are some ideas, some concepts we can consider in everything we do to perhaps bring a more digital and smart way of operating in the new world we live in. These are not innovative in themselves, we live with these ideas already. My intent is to try to capture these ideas and through the tactical implementation of these ideas bring real innovation.

Strategy – Strategy is not Keywords

Too often, we write our strategy or integration documents and load them up with keywords and slick powerpoint. We have to embrace that we touch real things. Don’t use stick figures and cartoons  – use real live examples, get your hands dirty using the systems on your job. Understand the details. Stop writing keywords on a white board and open the application!

Digital Strategy Keywords

The real idea is to be grounded in practical things, not abstract processes.

Open your schedule, look at the activity definition that exists. Open your material management system and see if the items can be tagged with schedule activities. Look into your progress measurement sheets and determine if you can insert scheduleID to smartly update the progress in your schedule (sound like a broken record!).

Strategy – Enter data once

This is talked about by everyone, although, we have all seen it – everyone has their own spreadsheet. Every group has their own systems: SAP, Contracts, Procurement, Engineering, Cost Control, Planning: I have illustrated in the past the issue with something as standard as “where is the master contract list”. Do we really have a strategy to enter data once? If so, really delve into the data people keep on their computers in their own spreadsheets. Try as best as we can to pull key reference data out of a source system

There exist systems that can facilitate this approach and allow for integration. Although be careful and understand that integration is not strategy. The implementations of strategy will likely require integration.

Data Integration vs Digital Strategy

Enter data once is integrated into the concept of data integration. Integration is not specifically the idea that “systems talk to each other”, integration is the ability to just smartly manage your job. Unless common keys exist between systems, you can’t hope to build that integration inside a black box your digital team will build.

A good example of this is the implementation of a web based cash flow

Web Based Cashflows – Sharepoint

Strategy – Be Visual

Project Controls teams are the masters of reporting. Being visual is not about (only) creating dashboards or complex databases the feed our dashboards. Being visual is doing exactly what we already do. Stay away from data tables – Long Live Death by S-Curves!

The tabular data presentation methods we use are antiquated. Looking at an engineering progress report with 50 rows and 20 columns of various EV metrics – too often that doesn’t answer me the question of whether or not I am on plan or not. That is what I meant to be visual. Look at the questions be want answers for, the core underlying questions and FOCUS. If the question is infact “I was to see the productivity factor for 50 different areas both weekly and cum”. Perhaps choose a dot-plot. Place the values all on a graph, perhaps scale the size of the dot on the budget hours, use a horizontal axis for % and the vertical axis to show the PF. That is what I mean to “Be Visual”.

Eng12

Strategy – Push processes and data ownership to the person actually responsible, not the “function” that is responsible

Payment certificates, progress measurement reports, monthly reports: the list is never ending. We have data that originates with contractors and a multitude of sources. However, too often, the sources of information do not have access to the real database  or word/excel file used by the project. If possible, structure your data sources such that owner can manage their data.

I will likely post a few case studies about this to get people thinking of possibilities.

 

Strategy – Use Agile management

  • Focus on Output not Documentation
  • Respond to Change as opposed to fixation on Original Plan
  • Focus on interactions between people, not underlying systems or tools

These topics can be very disruptive: Tell a company like Fluor or Bechtel to NOT focus on plans or procedures? Good luck. However, this is a disruptive day and age and we do need to take this approach. I have worked in this business for over 20 years and honestly, in general, project controls already (generally) follows these agile fundamentals. We have plans and systems and reams of documentation, but in the end, we do (informally) have to follow these simple strategic cultural elements

What is needed here is the firm support from the management team that a proactive view of the

“what is going to happen tomorrow”

is way more valuable that the concept of

“what was planned to happen tomorrow”

The concept that

“I need this new report now”

is more relevant than

“This is the report that is defined in the PPM”

AGILE management is not able sprints or standups. Its a culture to embrace a stance that while we have general goals and objectives, the focus needs to be on what are we doing today. The reason to have daily stand ups is to have clarity on what we are doing today. The reason to have sprints is to outline near term goals. The reason we have restrospectives is to analyze the difference between what we thought we were going to do, and what we did (and more importantly how to improve the next period). This is just good project management 101! The fact someone calls it AGILE, is not relevant.

JIRA in Construction

Strategy – Allow users to LINK to key fundamental data

This is aligned with “enter data once”. But, this is a core strategic ideal that needs to be championed. We will have data housed in source systems, but have we enabled the wider project team to access this information. That is the primary idea here.

Is information shared by way of excel exports that are distributed via email to various people at various times? Or, do we have a project where information is open and accessible. The latter is what our strategic approach to everything should be.

 

Strategy – Facilitate live real time data

Ultimately you will need some sort of approach to cadence and integration. Will we be using real time data, will we instead focus on cutoffs.

This can if we really delve deeper into what this really means, be very disruptive. With real time data, the need for monthly reporting is useless, the need for weekly reporting – is useless. Reporting is a natural output 24/7 from the way we operate. This cultural approach to construction management is where I think we need to go to be hyper reactive to change.

 

Strategy – Be Hyper Reactive

As I mention above, we want to have a core ideal that allows us to change course immediately when required. We have submitted the same schedule and cost reports every month – WHY? Do not be afraid to change things up, support a workplace where the team can be hyper reactive to respond to everything.

For schedule management, this requires the approach to insert activities into the schedule with daily vigor to reflect what is happening today.

For cost management this requires detailed ticket management approach to our tasks to allow visibility into everything we do 24/7

 

Conclusion – Do not be afraid to push boundaries

This list above is again just a beginning of a discussion. These are discussion topics that are occurring in every company around the world.

As we head into the future, we have an entire generation of new employees who are not afraid of technology. Quite the contrary, the newer generations of people will feel more comfortable updating a web form vs updating a word file. They will feel more comfortable posting progress updates to a social media site vs writing a daily report. They will be more comfortable making mistakes, and fixing them.

The dashboard craze is here to stay. The use of dashboards and the approach needed to create PowerBI or other application based analytics tick many of the strategic elements I have outlined here.

In the end – do not be afraid to push boundaries with your approach to construction management.

Major Milestone Tracking – By Darrin Kinney

A key quality of project controls management, is the communication of major milestones. The whole point is to review the changes of all the dates over time.

A key quality of project controls management, is the communication of major milestones. Every lead will have several spreadsheets with all our milestones listed on rows, and columns for the various interpretations (baseline, contract date, prior forecast, current forecast, contractor forecast, etc). So when we talk about dates, the difficulty is that everyone will have a different date in their mind. Continue reading “Major Milestone Tracking – By Darrin Kinney”