Hey, you development people

flipper35 · flipper35's Garage

Asking this for a friend because this project does not seem that difficult. How long would you take to complete the following project.

You have a flat file that gets dropped onto your server by another application. You have to determine if the file matches the hash. If so, send a good confirmation file. If not, send a corrupt confirmation file. You then have to parse the data and compare with a set of fields from your own invoice in SQL. If there is data missing, you send notification back to the sender what was missing. If the data is complete and it matches your invoice, send it for processing. If it doesn't match send it for review.

The original contract was under two months.

He was just recently thrown into this project and doesn't know what development tools, if any, the other company has. They should find that info out today.

The kicker? this is going on 16 months and so far the developer doesn't know why the data is parsed wrong, doesn't have the file verification automated and is no closer to a working product at this point.

For me, I know this could be done in a short time with a bunch of different tools from Visual Basic and SQL to Adobe ColdFusion.

The file verification would be easy but the developer does not plan to have that complete until it goes live. I have some opinions and none are good at this moment.

I don't understand why it would take so long. Maybe you can find out the flaw in the process that I don't understand?

flipper35 · flipper35's Garage

OK, so I asked these questions and the developer did not understand what I was asking.

"What development tools do they have available?
So, what tools or technology are they using to determine if a file has arrived and how to they verify the checksum or hash?
How do they create the file to tell the first place it is good, corrupt or missing info?
Are they parsing the info directly or trying to import the file into SQL?
Are they trying to do all this through SQL and views or stored procedures?
"

Ayles · 03-09-2021, 09:19 AM

We do this exact thing with an ETL tool call Xplenty...

https://try.xplenty.com/xp-paid-search-lp-solutions/?utm_campaign=2071745946&utm_source=google&utm_medium=cpc&utm_content=410990471705&utm_term=xplenty&adgroupid=79080289369&utm_campaign=FTF_Brand&utm_source=google&utm_medium=cpc&utm_content=410990471705&utm_term=79080289369&gclid=Cj0KCQiA1pyCBhCtARIsAHaY_5fUuq4EkfvouJrhTUGv jTeWgXVhSY2zG_b6r84vROWfAdJRJoYoAQMaApfyEALw_wcB

flipper35 · flipper35's Garage

Did it take 16 months to implement?

Ayles · 03-09-2021, 12:36 PM

Nope, i can spin up a new ETL in a matter of hours. Runs SQL queries, munges data, I can ingest a csv from s3 or FTP right into a databse, delivers automated failure messaging via email and slack. If a file from a client violates a duplicate key constraint I know as soon the job runs. Dont get me wrong, this takes SOME engineering resources but 16 months sounds crazy to me.

3rd_gear_Ted · 3rd_gear_Ted's Garage

Data from both side of the transactional process should only be able to be chosen from dynamic value lists built into the app. A drop down box with all the data choices is how you preclude this scenario.

flipper35 · flipper35's Garage

Ted, it is supposed to be all automated with no human intervention unless there is a data mismatch between the two invoices.

Ayles, can it deliver a file based message for the response?

What is frightening is the developer not understanding the questions asked to see where things are going wrong.

If it were me, I would fire that team and start over. I can't imagine something that simple wouldn't be further along inside 6 weeks, let alone 16 months.

Right now, they don't know how to check the file, they don't know why it rejects some and not others, they don't know what development tools they are using other than they do have SQL Studio with an Access database. Yes, Access that is accessed through SQL Studio.

id10t · 03-09-2021, 02:13 PM

Quote:

Originally Posted by Ayles

Nope, i can spin up a new ETL in a matter of hours. Runs SQL queries, munges data, I can ingest a csv from s3 or FTP right into a databse, delivers automated failure messaging via email and slack. If a file from a client violates a duplicate key constraint I know as soon the job runs. Dont get me wrong, this takes SOME engineering resources but 16 months sounds crazy to me.

Ayup. From two piles of bare metal I can set you up a server to accept the file and run the code, and set up a development machine and environment, and have working code to process the file in just a couple or three working days.

masraum · 03-09-2021, 02:14 PM

Quote:

Originally Posted by flipper35

Ted, it is supposed to be all automated with no human intervention unless there is a data mismatch between the two invoices.

Ayles, can it deliver a file based message for the response?

What is frightening is the developer not understanding the questions asked to see where things are going wrong.

If it were me, I would fire that team and start over. I can't imagine something that simple wouldn't be further along inside 6 weeks, let alone 16 months.

Right now, they don't know how to check the file, they don't know why it rejects some and not others, they don't know what development tools they are using other than they do have SQL Studio with an Access database. Yes, Access that is accessed through SQL Studio.

I was thinking the same thing. You don't seem to be talking to developers. Hell, I'm not a developer but even I've got ideas about how it would be done. I would think it could probably be done within bash or something like that (not that I'm a developer or suggesting that).

masraum · 03-09-2021, 02:15 PM

Quote:

Originally Posted by id10t

Ayup. From two piles of bare metal I can set you up a server to accept the file and run the code, and set up a development machine and environment, and have working code to process the file in just a couple or three working days.

LOL! So what you're saying is 16 months sounds about right?

id10t · 03-09-2021, 02:16 PM

I have a feeling since Access is involved there is no one with real development skills and training.

Dev tools the other company has? Who cares. Determine data exchange method (sftp, http, etc? pull or push?, etc) and the format for the file(s) sent back and forth. If Company B is happy using a knitting needle and magnet to make 1 and 0s on their end, that isn't my problem as a dev on "our" end.

Ayles · 02:38 PM

Quote:

Originally Posted by flipper35

Ted, it is supposed to be all automated with no human intervention unless there is a data mismatch between the two invoices.

Ayles, can it deliver a file based message for the response?

What is frightening is the developer not understanding the questions asked to see where things are going wrong.

If it were me, I would fire that team and start over. I can't imagine something that simple wouldn't be further along inside 6 weeks, let alone 16 months.

Right now, they don't know how to check the file, they don't know why it rejects some and not others, they don't know what development tools they are using other than they do have SQL Studio with an Access database. Yes, Access that is accessed through SQL Studio.

I am sure this can be done though I have not explored extracting the error messages. I just read them in place to see where the fix needs to happen.

Tishabet · 07:12 PM

To add here, a lot of complexity (or lack thereof) could be involved with this flat file and how well structured the data is and it's unclear what the situation is from your post. If you're talking about well-structured and consistent data then this should be a relatively straightforward thing to develop using off-the-shelf software as others have noted.

I built a system around 15 years ago which parsed flat dumps of data for a big bank (you would recognize their name), millions of lines per file. Between the sheer volume of data (and building something low-latency so that we could parse/handle it all before the next one got dropped) and the inherent grunginess of the data (data included lots of human input info that was poorly sanitized) it took months to get it working well.

HardDrive · 03-09-2021, 03:59 PM

Oh for god sake. I'm a delivery manager at a large financial institution. 6 weeks max including dealing with ALL deployment and authorization issues.

16 months...sweet lord, fire that SOB and sue to get your money back.

cstreit · cstreit's Garage

Quote:

Originally Posted by HardDrive

Oh for god sake. I'm a delivery manager at a large financial institution. 6 weeks max including dealing with ALL deployment and authorization issues.

16 months...sweet lord, fire that SOB and sue to get your money back.

This. I would say a few weeks for the actual work and the rest for hardening it for production.

flipper35 · flipper35's Garage

Quote:

Originally Posted by id10t

I have a feeling since Access is involved there is no one with real development skills and training.

Dev tools the other company has? Who cares. Determine data exchange method (sftp, http, etc? pull or push?, etc) and the format for the file(s) sent back and forth. If Company B is happy using a knitting needle and magnet to make 1 and 0s on their end, that isn't my problem as a dev on "our" end.

When I said the other company, it was a merger so I apologize I was not clear there. They were asking what tools this developer was using since they were not producing results and they wanted to know if they even knew what they were doing, which everyone pretty much knows they don't.

The rest is not a direct reply.

The flat file comes from the other company in the same format and layout every time. If it isn't, it is supposed to be sent back.

Last night I found out this has been 16 months since the merger and the other company that was merged had been through over a dozen rounds of testing prior to the merger so this project has been going on well over 2.5 years.

For full disclosure, I have been in IT for decades so I was pretty sure this wasn't a big project but I wanted to make sure it wasn't just me. I am trying not to be biased here, but it is really hard.

I did offer to send some code examples to help them out. I did send my friend some examples as they are not really an IT person and I wanted to help them understand some of what was involved. They gave the opinion to the management prior to looking at the code that this project cannot be completed by the current team. After reviewing the code they said they would be confident the team could do it except for the lack of talent, knowledge and skill.

flipper35 · flipper35's Garage

That flat file has less than 100 lines.

3rd_gear_Ted · 3rd_gear_Ted's Garage

CMMI level 3 - document minimums , or its not really a SW project

User requirement specifications - user interfaces - work processes

System Architecture - key issues - owner principle stories

Implementation Strategy - quality criteria - unit task specs -unit test cases

Pre release defects are on YOU

Post release defects are on the OWNER

What so hard???

id10t · 03-10-2021, 06:17 AM

Quote:

Originally Posted by 3rd_gear_Ted

CMMI level 3 - document minimums , or its not really a SW project

User requirement specifications - user interfaces - work processes

System Architecture - key issues - owner principle stories

Implementation Strategy - quality criteria - unit task specs -unit test cases

Pre release defects are on YOU

Post release defects are on the OWNER

What so hard???

I would kill for specs like that at times, other times I'm happy for the freedom I have

My work tickets are usually like -

"The other college in town needs to give us a list of student ids to mark as being accepted into/graduated from their program for a certain term/year. Make it happen"

or

"HR needs a dynamic PDF created when someone completes a training course"

While those types of tickets are easy (and the OPs project sounds like one of these - "retrieve file from $location, parse, validate, and store") due to the freedom I have in how to actually solve the problem, it sucks when tickets are like

"Update and rewrite the transient student system from our 25 year old tech stack to the new hotness" just due to the technologies involved in the current implementation (IBM's VG/EGL anyone?) but also due to the lack of documents on what the state expects as a REST api or what the state provdes as a API - apparently there is some magical gateway service that when we are done we'll be able to map the two systems together....

Ayles · 03-10-2021, 08:17 AM

Quote:

Originally Posted by flipper35

That flat file has less than 100 lines.

The file only has 100 rows? Always???

This could be done in Python or R pretty quickly.