Data Transformation Archives - Management Information Analysis

Make Your ETL Project Lean and Clean with Clear Objectives and the Right Skills.

October 19, 2016 by MIAWebAdmin

As you know, when you apply ETL to your data transformation project, data is “extracted” from the data sources using a data extraction tool. It is then transformed using a series of transformation routines. This transformation process is largely dictated by the data format of the output. Data quality and integrity checking are performed as part of the transformation process in the data staging area. Finally, once the data is in the target format, it is then loaded into the data warehouse ready for presentation/reporting.

The process is often designed from the end backwards in that the required output is designed first. In so doing, this informs exactly what data is required from the source. The routines designed and developed to implement the process are written specifically for the purpose of achieving the desired output, and only the data required for the output is included in the extraction process.

Business rules that define how aggregations are achieved and the relationships between the various entities in both the source and target are designed and therefore coded into the routines that implement the ETL process. This approach leads to tight dependencies in the routines at each stage of the process.

ETL makes sense when the target is a high-end data engine, such as a data appliance, Hadoop cluster, or cloud installation. The ETL approach can provide drastically better performance in certain scenarios if a proper ETL pipeline is designed, allowing parallel execution for faster performance.

The training and development costs of ETL need to be weighed against the need for better performance as well. So your research. Ask a lot of questions. Engage a successful, seasoned team of data transformation, integration, and planning analysts. We cost less and deliver more.

Our Austin, Texas location keeps us at the epicenter of planning analytics and data-driven healthcare innovation and keeps our global capabilities and services at the forefront of our industry.

Email us at kcerny@mia-consulting.com or call us at 512.478.3848 to start a friendly, productive conversation.

Data: Transform It. Own It. Apply It in Your Custom Applications.

September 16, 2016 by MIAWebAdmin

Your own solution:

Every company has unique business needs. As a result of co-working with a custom software developer, you become the owner of applications that meet all your requirements. Companies often adapt their practices to how the applications work. You need to do the opposite. Custom software development services help you remain and become more competitive. It also has a major impact on productivity since the package of software is tailored to your needs. You no longer have to “share” the same solutions with others.

Support and updates:

With custom applications, your work is not dependent on the provider’s development cycle. You are in charge of all the updates and improvements. Working with in-house developers is only more efficient when your analysts do not have competing priorities. You have more personalized support using your outsourced partner. The team is aware of your individual requirements. As well, you have fewer troubles with hardware environment for applications’ hosting. Custom solutions do not need specific hosting. Applications are adapted to your existing server environment. You no longer have to pay twice as much for having twice as less.

Remember, in-house developers’ reputation and future business is riding on how successfully they develop your custom applications.

Virtually no security issues:

Using packaged software is risky. If someone hacks the software, all the companies using it are in danger of losing proprietary information. Custom software development teams provide their clients with applications that may only be affected by targeted attacks. Your application needs to have extra security measures to ensure its safety.

A custom data management application partner may also enhance your solution’s integration with other software. It makes your team’s work much more intuitive and easy.

Step-by-step production:

Custom software can be produced in chunks. It means that you can run MVP versions several times to check if there is a tight fit to your business and its data management needs. You may add or alter certain parts. Professional developers will help you achieve high performance of your application.

Every project is unique and has its own requirements. If your choice is based on certain needs and budget, you will be able to come up with the right decision. Even so, we can assist you with best practices information and processes. You need to know the differences between development teams. Also, you should be aware of proper requirement analysis.

These guidelines will be almost enough. You need to define what kind of business problems your application will be solving. We need to be especially clear that this step is essential for young companies. You indeed know that you have to work with the right people to achieve success.

Our data integration and custom application development expertise allows our analysts to load, transform, mix, and match data sets as needed. We can blend internal with external data sources to establish historical trends and to project need and demand, which are the backbone of any feasibility study.

Now that we have some clarity on process and really positive business outcomes, call us for a friendly conversation about your data and how we can elevate it to make really savvy business decisions and plans. We’re easy to talk to and we listen well. 512.478.3848.

Or, you can share your thoughts and success stories by emailing us at kcerny@mia-consulting.com. We’ll share too.

A Little Data Transformation Recap with Big Implications for You.

September 7, 2016 by MIAWebAdmin

If CIOs can convince their business unit leaders to implement integrated data platforms successfully, this solution can do the following:

Make both structured and unstructured data available and fully integrated to provide a 360-degree view of the customer
Reduce the cost and streamline the process of storing cold data (archived) and warm data (not required for real-time or data that is accessed often) in economical yet responsive solutions, and still have it available for analytics
Turn poor decisions, made on the basis of haphazard guesswork, into well thought-out and successful results that improve operational efficiencies, drive revenue, and provide a competitive advantage in the marketplace
Drive significant bottom-line improvements through the automation of routine decisions
Give organizations the ability to hold exponentially increasing amounts of new, mostly ad hoc, unstructured data sources while providing the flexibility to adapt to a complex, ever-changing business and data environment

Data is a critical asset of any company, and the success of every company lies in turning this data into insights upon which future plans can be molded. With the help of a properly prepared business data vision, backed up with leadership and support from the C-suite, the CIO can create the framework for an effective data management road map and a technology infrastructure that guide projects in the implementation to achieve their business objectives.

By adopting a blended approach that integrates legacy data with emerging big data and enabling analytics across the entire organizational data platform, the CIO becomes a leader who helps organizations achieve true, value-added, integrated, end-to-end business intelligence.

Give your thoughts and success stories in an email at kcerny@mia-consulting.com. We’ll share too.

Let’s Define Data Transformation and How It Elevates Your Data’s Value.

August 18, 2016 by MIAWebAdmin

Data Transformation Services, or DTS, is a set of objects and utilities to allow the automation of extract, transform and load operations to or from a database. The objects are DTS packages and their components, and the utilities are called DTS tools. DTS was included with earlier versions of Microsoft SQL Server, and was almost always used with SQL Server databases, although it could be used independently with other databases.

DTS allows data to be transformed and loaded from heterogeneous sources using OLE DB, ODBC, or text-only files, into any supported database. DTS can also allow automation of data import or transformation on a scheduled basis, and it can perform additional functions such as FTPing files and executing external programs. In addition, DTS provides an alternative method of version control and backup for packages when used in conjunction with a version control system, such as Microsoft Visual SourceSafe.

DTS has been superseded by SQL Server Integration Services in later releases of Microsoft SQL Server, though there was some backwards compatibility and ability to run DTS packages in the new SSIS for a time.

The DTS Designer is a graphical tool used to build complex DTS Packages with workflows and event-driven logic. DTS Designer can also be used to edit and customize DTS Packages created with the DTS wizard.
Each connection and task in DTS Designer is shown with a specific icon. These icons are joined with precedence constraints, which specify the order and requirements for tasks to be run. One task may run, for instance, only if another task succeeds (or fails). Other tasks may run concurrently.
The DTS Designer has been criticized for having unusual quirks and limitations, such as the inability to visually copy and paste multiple tasks at one time. Many of these shortcomings have been overcome in SQL Server Integration Services, DTS’s successor.

Data integration is the process of combining data from many different sources into an application. You need to deliver the right data in the right format at the right timeframe to fuel great analytics and business processes.

Our data integration and custom application development expertise allows our analysts to load, transform, mix and match data sets as needed. We can blend internal with external data sources to establish historical trends and to project need and demand, which are the backbone of any feasibility study.

Now that we have some clarity on process and outcomes, call us at 512.478.3848 or email us at kcerny@mia-consulting.com for a friendly conversation about your data and how we can elevate it to help you make really savvy business decisions and plans.

The Data Transformation Challenge Is Here in Austin – Right Now!

August 12, 2016 by MIAWebAdmin

One of our Austin Technology and clean-energy movers is CLEAResult. They teach their clients energy usage best practices and new techniques. Just last week, they posted a position opening for an ETL analyst.

This makes sense in the context of their recent wave of acquisitions and the need to get data out of those spaces and into the dashboards that impact what they offer clients. And there’s more.

A Computerworld article discusses a trend of increasing demand for ETL and other data transformation analysts and data scientists in the Austin and Southwest areas. Here is one salient comment from their coverage of what’s happening.

“Local recruiters are looking more and more for people in the world of mobile applications, and people with good skills in data integration, big data and . . . visualization and data analytics,” Jake Dominguez, CIO at Austin-based chipmaker Advanced Micro Devices stated. “There is fierce competition for design engineers, for manufacturing and new technology, and we are seeing a lot of design centers being built.”

Already one in eight jobs in Austin is within the technology sector, and since those jobs pay well, they account for a quarter of the local payroll, says Drew Scheberle, vice president of the Greater Austin Chamber of Commerce. The chamber estimates that there are about 4,700 technology companies in the Austin area and that 150 people are moving to the area every day.

The demand is much greater than the supply of proven ETL and data integration talent available in our area. You need to get your data to work right now, so what can you do?

Additionally, we can help you monitor trends, measure progress toward a specific quantitative goal, or flag shifts in your market or business operations.

Engage a successful, seasoned team of data integration analysts. We cost less and deliver more. Our Austin, Texas location keeps us at the epicenter of planning analytics and data-driven healthcare innovation and keeps our global capabilities and services at the forefront of our industry.

Email us at kcerny@mia-consulting.com or call us at 512.478.3848 to start a friendly, productive conversation.

Mobile Apps Are All the Buzz, but Not for All

June 30, 2016 by MIAWebAdmin

Despite the much-publicized demand for mobile app development in the enterprise, spending for such initiatives remains low, according to a new report from research firm Gartner, Inc. that parallels the findings of many other studies.

Gartner’s earlier report (late 2015) contends that, “Organizations want to have full control over their mobile app development initiatives; however, maintaining a pure, in-house development environment is difficult to achieve given mobile is a relatively new competency to many developers.”

The report also states that mobile app development “entails many complexities and specific activities, such as UX design and psychology or cellular coverage testing, which may be more efficiently handled by an outsourced third party experienced in mobile app development.”

Despite the hype surrounding mobile and opportunities for custom apps to enable greater workforce productivity and enhanced customer engagement, the number of custom apps created is low.

Here’s a quick and easy way to visualize the difference between the “buzz” and the results:

On the other hand, virtually all knowledge workers are using more data science- and coding-type activities in the mainstream work cycle than before, and many more are increasingly engaged in digital transformation as stakeholders in agile teams or business product managers.

In the long run, more of the population will eventually engage in a level of app construction skills but for that to materially change, schools have to do a better job teaching app building.

Meanwhile, our teams can be a highly productive bridge resource to get your custom applications mapped out, prioritized, and running– on time and within budget.

Make sure that you are not hung up in the hiring loop looking for ETL and data transformation services when you can have a trusted, security-robust partner working to get you on top of your critical data.

You can bring your data flow to its highest and best uses or you can let it languish while you guess and your competitors pass you on the back stretch. And you DO NOT have to bet the farm and the horses to get where you want to go with your data assets.

Our Austin, Texas location keeps us at the epicenter of planning analytics and data-driven innovation and keeps our global capabilities and services at the forefront of our industry.

Email us at kcerny@mia-consulting.com or call us at 512.478.3848 to start a friendly, productive conversation.

Data Flow and Transformation Snapshots Mid-Year 2016

June 23, 2016 by MIAWebAdmin

The Internet of Anything (IoAT) creates a new paradigm that requires new thinking and new data management systems, and these solutions will mature and permeate enterprises in 2016.

There is a market need to simplify big data technologies, and opportunities for this exist at all levels: technical, consumption, and so on. In 2016’s second half, there will be significant progress toward simplification. No matter who you are — a cluster operator, a security administrator, or a data analyst—we all want Hadoop and related big data technologies to be straightforward.

Here’s one more for Hadoop: In 2016, Hadoop will be used to deliver more mission critical workloads beyond the “web scale” companies. While companies like Yahoo, Spotify, and TrueCar all have built businesses which significantly leverage Hadoop, we will see Hadoop used by more traditional enterprises to extract valuable insights from the vast quantity of data under management and deliver net, new-mission, critical analytic applications that simply aren’t possible without Hadoop.

Security will grind its way to the top of your priorities list. Europe recently set strict regulations around data, which means that organizations will need to be strategic about how they’re tackling data security issues.

Rather than considering data privacy an afterthought, people will need to be proactive in explaining exactly how they will use the data and ensuring compliance with local and global laws. This is particularly true with the large increase in accessing data remotely (via mobile devices).

We are a good match to be your analytics thought partner and trusted virtual team that delivers your data to your dashboards.

Our Austin, Texas location keeps us at the epicenter of planning analytics and data-driven innovation and keeps our global capabilities and services at the forefront of our industry.

Email us at kcerny@mia-consulting.com or call us at 512-478-3848 to start a friendly, productive conversation.

Get Your Data Out of Those Hobbit Holes!

May 27, 2016 by MIAWebAdmin

Whether it’s revenue-cycle management in healthcare delivery or back-office forms administration in financial services sectors, the process of getting usable data out of desktop silos and into secure cloud warehouse continues to creep along. But really, folks, it must get handled.

The shared anxiety around data risk will unify leadership and IT teams as the new crop of solutions enable simpler administration, transparent governance, and agile collaboration.

The movement will be further accelerated as more entities are seeing elastic cloud architectures as the mainstay of big data and intelligence standardization decisions. In fact, this will become the litmus test for Chief Data Officers and CTOs: if the solution doesn’t have a built-in elastic architecture, it won’t be fully deployed.

Here’s another thing: in general, we will see a major movement to adopt cloud data storage in the financial service industry. In fact, one of the top five financial institutions will announce a cloud-first or cloud-only IT philosophy –adopting one or more of the big three players.

If the CIA is doing it, why can’t the banks?

Adapt, die, or outsource?

Data preparation will be a critical capability of subject matter experts who, traditionally, relied on others to get data ready for them. In order to transform data into information on demand, people doing risk analysis, customer targeting, security monitoring, marketing, or sales operations will need the necessary skills and tools to handle self-service data preparation at scale.

Across all of these changes in how we manage our data assets, we see outsourcing the ETL process and teaming with talented analyst groups as practices that will cost-effectively accelerate your big data and analytics priorities. Outsourcing is now mainstream.

Our Austin, Texas location keeps us at the epicenter of software engineering innovation and keeps Management Information Analysis’ global capabilities at the forefront of our industry.

Call us to start a friendly and productive conversation. 512-478-3848

That’s all for today, but please stay tuned to this Management Information Analysis and our Planning Analytics blog channels for more actionable insights.

Meet the Future of ETL and Custom Applications

May 20, 2016 by MIAWebAdmin

We’re now into the meat of 2016, and if you take a look around our business and healthcare ecosystems, you’ll see that a number of positive changes are accelerating across the custom applications development of our information management worlds.

Here is some of the information we have parsed from trusted and knowledgeable sources:

Companies are adopting NoSQL policy for their critical operations and cloud computing their data in 2016. The era of big data is upon us, with Windows 10 being a predecessor of an online OS (or a beta version, which will be massively updated in the future).
- Android is preparing their counterpart for Win 10 based on Chrome and Spark, conquering new horizons as cloud hosting becomes more functional and popular each day.
Software future trends are firmly within the cloud. Just analyze the job listings: whenever you see any big company involved in software-development hiring, they say, “cloud experience would be an advantage.”
- We assume you will agree with the statement that 80 percent of resources (be it time, money, or developer’s effort) spent on ANY new custom application is spent on ETL process. Regardless of the coding language that the app is built on and the trends that software follows, you have to gather the raw data, transform it into operable information, and load it into the database so your custom applications can process it.
Current software development trends explicitly show that current data analysis systems will not cope with big data efficiently enough. Issues with scalability, simultaneous processing of different data sources, and efficient data transformation pose big threats and require an efficient solution strategy. One possible solution might be Red Hat’s data virtualization technology.
- Therefore, the company that will show the most efficient way of overcoming the ETL issues can be forming new software trends in big data for the years to come.

Add to this the possibility of packing it in containers, thus unifying the tools and making the app a multipurpose cross-platformer, and only the sky will be the limit.

Our Austin, Texas location keeps us at the epicenter of software engineering innovation and keeps Management Information Analysis’ global capabilities at the forefront of our industry.

Call us to start a friendly and productive conversation. 512-478-3848.

Do Anything With Your Data – But Please Protect It

May 13, 2016 by MIAWebAdmin

It’s May. Is it time to ask about outsourcing your ETL processes?

An automated ETL tool typically provides a visual environment or platform that lets you see large parts of the ETL process (e.g. where the data is coming from and going, which calculations are performed on it, etc.) in ways that old code practices—however carefully crafted and documented—will never be able to replicate.

As we have offered many times over the past 18 months, the sooner and more efficiently you get your data out of silos, the better able you are to make analytical and predictive uses of your data. Now that always leads to one of the biggest benefits of big data.

Here are a few big data benefits:

Predictive modeling, which involves running simulations to predict the outcome of altering variables in a dataset, is one of those rewards at the end of your data rainbow.

With huge amounts of data from many different data points, the relationships between different variables can be examined in detail. This has proven advantageous in many circumstances, where analysts have found that tweaking a certain variable had unexpected consequences or outcomes.

Big data can highlight dangers looming on the horizon by showing what the likely consequences of a path of action will be. This has extended into the physical world, where machinery fitted with sensors that constantly feed data on their operations are becoming widespread across the industry.

Your big data systems can even build up an extensive understanding of how a machine works and when it is likely to go wrong, meaning that repairs can be scheduled, reducing costly downtime to a minimum. Flaws in the design or manufacturing process can be identified at an early stage.

Data always contains surprises. Tailor transformations to any event or field with our custom ETL Plus service. Explore your data in real-time, write alerts, and get notifications. Get your data out of silos, off other servers, and into your agile data management and planning streams.

And protect it.

As mobile computing advances, information management leadership will be called upon to safeguard equipment, data, and applications that employees will be operating at any time and from anywhere.

To accomplish this, your analysts must implement security systems for mobile and IoT applications that must be agile, robust, and able to function within, at, and beyond the edge of the enterprise network. Equally important will be aggressive policy development and training of employees, who can often be a leading source of mobile computing security abuse.

These devices are used externally as well as internally within companies. With the amount of time that a mobile device is used off premises, risks of device loss, data loss, or security breaches escalate.

So, too, does the risk of devices being misused by those who are both authorized and unauthorized to use them. Latch on to a partner who can accelerate your data transformation, making your data visual, usable, and immediately more agile!

Our Austin, Texas location keeps us at the epicenter of software engineering innovation and keeps Management Information Analysis’ global capabilities at the forefront of our industry.

Call us to start a friendly and productive conversation. 512-478-3848.