Why Data Integrations Fail: Closing the Gap between Business and IT, Reducing Cost & Complexity in the Cloud by datasamurai.io is a product of Tenor Enterprises, LLC ©2014 Why Data Integrations Fail: Closing the Gap between Business and IT, Reducing Cost & Complexity in the Cloud by Introduction ............................................................................................................ page 1 Measuring Success ................................................................................................ page 3 Bigger is not Always Better ...................................................................................... page 3 The Custom Job ..................................................................................................... page 5 Meeting your Needs ............................................................................................... page 6 Thinking in the Cloud .............................................................................................. page 7 Success, Every Time ............................................................................................... page 8 Introducing datasamurai.io ...................................................................................... page 9 Case Study: Sales Leads to CRM .......................................................................... page 10 Introduction of Change ........................................................................................ page 11 A Solution .............................................................................................................. page 12 Contact Information ........................................................................................Inside Cover Table of Contents Why Data Integrations Fail: Closing the Gap between Business and IT, Reducing Cost & Complexity in the Cloud Introduction Let’s face it, data and systems integration is diﬃcult, complex and often times a daunting task. There are a myriad of providers and solutions out there that solve this very problem; however the typical solution is often just as complex, expensive and diﬃcult to maintain than trying to do it all yourself. What’s worse is that most vendors and consultants that provide the type of knowledge and experience in delivering solutions using these data integration platforms are expensive, integrations take a long time and you’re still left with a heavy support bill or face trying to ﬁnd resources yourself to help maintain things as they break change. How do you ensure your data integration needs are going to be met? How do you ensure it will be easy to implement, maintain and remain ﬂexible during times of change? One of the oldest problems in IT has been getting systems and/or data integrated together. How do I get system A to talk to system B and vice versa? How do I make informed business decisions on my data when it’s scattered across apps, databases and the cloud? How do I get a holistic and complete view of a customer when it’s split between my CRM, post-sale in-house support applications, ERP and possibly dozens more cloud or conventional systems, each used by a diﬀerent department, partner or vendor? 98% of data integration projects that are considered a failure are not because of technical challenges or overrun budgets, but because they either a) didn’t actually integrate everything in a way that was useful, b) only provided a partial solution, or c) something changed shortly after go-live that caused a brittle or fragile data transfer process to break. 1 It’s no secret that more and more organizations are moving from traditional software and services to cloud based solutions, SaaS, IaaS and PaaS oﬀerings. Using smaller, less expensive and targeted cloud based solutions and tools to accomplish speciﬁc tasks and goals not only makes sense, they also eliminate a lot of IT costs and tied-up resources. With less infrastructure, servers and software to support and manage, IT teams can focus on improving the business in other ways, support or enhance internal systems or even reduce consulting and contracting fees. These also give business professionals and management the opportunity to make decisions based on their needs as opposed to what IT thinks is the right solution for them. Would you let your auto mechanic make all of your investment portfolio decisions (not that your mechanic wouldn’t have insight into the market)? SaaS PaaS IaaS 2 Measuring Success In terms of any technology investment there are some typical key factors to consider when making the decision: 1. What are the speciﬁc needs of my organization? 2. What are the short term implementation and long term support costs? 3. What is the ROI? Often times we tend to look at this list in reverse order. While each of these are equally important factors to consider, often times what the needs of the organization are get swept away by the other two which speak to immediate and longer term budgets, costs and resources. Bigger is not Always Better Some of the largest, most commonly used ETL tools (stands for Extract, Transform & Load), are also the most ﬂexible, extensible and expensive approaches to solving the problem of connecting data. Most tools attempt to be everything to everybody, and, if you’re a very large fortune 50 or fortune 100 company with 50k+ employees, global operations and customers, partners and vendors sending and receiving data, with an IT budget of $500M+ per year, using these tools and staﬃng a team to maintain, develop and create solutions with them typically wouldn’t cause much hesitation on the decision. Because these platforms require custom programming to actually make them do anything of use, some even having their own programming languages, terminology and even their own operating systems and sub-systems which require years of training and experience to know how to use and be proﬁcient in, they are typically out of reach to most small to mid-size companies; the professionals who have this knowledge are also some of the most expensive 3 Bigger is not Always Better, cont. resources in the IT industry. What if you have a small or mid-sized business, or are part of a larger organization trying to reduce costs and build in proﬁciencies? Would a small company with 500 employees purchase a full-blown Oracle HRIS system, SAP and a custom implementation of PeopleSoft? No, of course not; that would be overkill. There should also be no reason to do that with data integration; ever. When larger or more complex systems are purchased and implemented to solve data integration problems, often times it is a failure from the cost and ROI perspective. One of these systems can cost from $250k up to $1M+ to implement and take live. From there, maintenance to keep it running, hardware or hosting, etc. can average over $50k per month. That’s a lot of money and typically not worth it just to get a bunch of data ﬁles into your system on a daily, weekly or monthly basis. Showing real cost beneﬁt it is diﬃcult there, but if the amount spent on software licenses, hardware and resources to get a customer or partner integrated is taking any signiﬁcant percentage of a customer’s or partner’s revenue stream for your company, there are probably better ways to accomplish that same task in a fraction $ Pe o $ ple $ So ft $ of the cost with more ﬂexibility and faster time to market. r O e l c a $ 4 The Custom Job Other approaches to data integration include custom development to cover a few small targeted integration that the business or IT feel are necessary enough to warrant the cost. What happens when something down the road changes, or new or diﬀerent data is needed? Approaching custom software development with an eye to the future and change is a challenge for most; not to mention often the most expensive and time-consuming approach to software development. This is typically prohibited by limited time and budgets to complete this type of work. Also, custom software development is not an exact science and takes time to implement, time to make changes and has real dollars associated with every change, feature or bit of data you need to move through it. Whether you’re building it in house or farming the work out to a consulting ﬁrm you get just that, a system that ends up being brittle and diﬃcult to maintain. You end up having to wait days, weeks or even months to get changes implemented that the business needed immediately to support its sales or operations; not only costing the business through IT staﬀ and resources, but potentially lost revenue and eﬃciency or lack of business insight. 5 ! Meeting your Needs Deﬁning your exact expectations for a data integration project is challenging. More than likely no one person in the organization understands the full domain of knowledge and data that is located through the company. A CEO or VP may know the types of ﬁgures they want to see, an operations manager may know what customer information they need, but where is it all located? Who has it and controls access to it? Is it in a system and/or format that’s usable for reporting or other processes? One day you may need to see a full view of orders through initial purchase order all the way through delivery and customer satisfaction survey results. The next you may not care as much about that because the company’s focus has changed to pre-sale marketing eﬀorts and you need to concentrate on both the prior regarding orders and a newly invigorated focus on marketing campaign response rates through the sale of a product. Typically though this data is scattered across multiple systems. And while you may have already integrated your CRM and order fulﬁllment tool, perhaps the only thing its sharing are orders, and now you need lead and campaign response data as well. Being able to not only meet today’s needs but also to adapt is extremely important. If your data integration strategy relies too heavily on others, e.g. IT, support teams, consulting agencies, vendors, etc., you’re going to miss out on opportunities as they present themselves and that can add up to real losses and missed revenue. 6 Thinking in the Cloud Cloud computing, services and big data are popping up all over the place with the goal of helping business grow, expand and stay ﬂexible in the face of change. Most of these services oﬀer aﬀordable alternatives to owning, maintaining and building internal infrastructure and applications; why not data integration? Quite a few services in the cloud already exist out there that help tackle the problem of data integration; however they all work under a speciﬁc concept, you connect 1 system and 1 data item type (like a customer) and you send to 1 other system, that’s it, you manage 10’s, 100’s or 1,000’s of 1:1 connections for every single possible combination of data you can think of. This often starts out simple (e.g. our order example from before) but can quickly become unwieldy and cumbersome to maintain and end up being counterproductive when change is introduced later down the road. The upside to most of these is that they can be conﬁgured by a typical business user, not an IT guy. ? Other services are a bit more complex; often resembling a workﬂow diagram in the cloud with complex data ﬂows, transforms, etc. This can also be cumbersome and need highly technical and skilled people to maintain and you’re right back to the traditional ETL style tools, only in the cloud. 7 Success, Every Time In order to meet all of the challenges it’s important to ﬁnd a solution that ﬁts all 3 major aspects of a successful data integration project, as well as provides the ﬂexibility necessary into the future to quickly adapt to change. Furthermore, a solution that leaves out a lot of the complexity, gives you a holistic view of your data and provides advanced analytics, data validation, cleansing and normalization, value-add services such as address, phone and name validation, geographical location lookups, weather data, census data, etc. When changes are needed, they shouldn’t break things, they should simply replicate through, expand capability and be easy enough to manage and integrate with existing processes. Add a new ﬁeld, how about a new type of customer or order, integrating a new system; all of it should be a cinch without adding complexity. Finally, almost everything you do with the write tool should be able to be done by someone who knows your business, not just about computers, software or data. 10 1 10 0 8 Introducing datasamurai.io datasamurai.io provides everything you need to be successful in your next and all future data integration projects. Unlike every other data integration platform out there, datasamurai.io gives you a central repository that allows you to deﬁne a single picture of your data; from there you can bring data in and export data back out all through a single link for each place you want data in or out of. Need to hook up your cloud based CRM, in house Oracle database, 3rd party vendor’s API and a torrent of ﬂat ﬁles coming at you through FTP, HTTPS and SFTP? Easy with datasamuri.io. Want to see all of that data together and be able to easily push updates from system to all of the other ones from a single action? datasamuri.io makes it simple. Need CASS address standardization and validation, phone number lookups, email blacklist and do not call list validation? With datasamuri.io consider it done. datasamuri.io is your answer to all of these data challenges and more. With the capability to pull data into and export data out of your repository to and from almost any source imaginable with more being built every day, datasamurai.io allows you to point and click your way to data integration utopia in a matter of minutes. Use a cloud based CRM, how about ticketing and support system, have in-house SQL, Oracle, MySQL or PostgreSQL databases, Excel spreadsheets, ﬂat or comma delimited ﬁles, XML ﬁles, web services, APIs or just about any other data source imaginable that you need to integrate? We have what you need to do it, and if we don’t; building it is our highest priority. 9 A way too common scenario... A small 300 employee business services company which performs outbound appointment setting calls on behalf of their customers oﬀers various value-add services, among them is the generation of sales leads for their customer’s vertical market, or the ability for their clients to send data on a daily basis with leads and potential customers. The scenarios for regular data feeds are handled by IT which consists of a single developer, Bob, and a small team of network, desktop, telephony and support analysts. Leads are taken from the customer, loaded into a home-grown on premise SQL Server database for the outbound contact management team, and are also uploaded into a CRM tool, in this case Salesforce.com which is used by the sales and account reps for each client account. Bob has to create a “job” for each customer, which is a custom software component, to move the customer’s data into the database as well as upload it to the CRM. Both take diﬀerent formats and therefore require each customers’ data to be massaged and changed to meet the needs of each system. That comes out to at least 2 jobs for every customer and a current load of 42 jobs that run on several diﬀerent schedules. Bob manages all of the customer expectations in a spreadsheet which he manually reviews each day. Each new customer takes about 2 weeks to get set up. 10 Introduction of Change One of the larger customers of this company recently asked that all potential appointment leads ﬁrst have their addresses and phone numbers validated to help reduce wasted time and costs on their per dialed number payment terms. Without a means to provide this type of pre-validation and data cleansing, the company was left scrambling to try and ﬁnd a solution to the problem. To add insult to injury, Bob was already spending upwards of 6 hours a day, 6 days a week maintaining, correcting and monitoring the existing data jobs the organization had. Hiring another developer for that not only was necessary, but it would take weeks to get them started, trained and up to speed being productive. By that time the customer would have found another vendor as they were already unhappy with the costs associated with not performing address and phone number validation up front. 11 A Solution After a long search for a solution to this mounting problem, the marketing director found datasamurai.io. datasamurai.io was able to provide not only automated import and export of the data coming in from customers, combine and aggregate leads and push them simultaneously to their outbound calls system but also directly into their CRM, they also were able to use data appends to cleanse and validate address and phone data for those leads not only for their current customer demanding it, but also other customers to proactively show cost savings and beneﬁts; which increased their own sales and demand/share of outbound business services by 38% since implementation. The data imports and exports are all handled by the marketing management team in charge of customer relationships and onboarding and no longer the sole responsibility of IT as business owners are able to conﬁgure imports, exports and value-added data append services with a few easy clicks. Bob is now able to spend his time consulting with customers’ and custom needs, supporting the outbound call team’s own internal software and gets his Saturday’s oﬀ. 13 For More Information Contact: Tenor Enterprises, LLC | 866.868.5957 sales@tenorent.com | support@tenorent.com PO Box 290841 | Port Orange, FL 32129-0841 www.tenorent.com Tenor | ten-or | (tnr) noun: continuous course, progress, or movement Have a nice day.