Always the same data in the same format with the same number of rows. If not, it is supposed to get rejected and tell the sender why. So if row 33 is missing it gets sent back saying row 33 is missing. Very complex stuff, I know.

As a former software engineer/consultant/solution architect myself I usually take the developer's side on these things... I've definitely had to have many talks with clients and stakeholders etc to explain to them why their seemingly simple request is not simple at all to implement.

That being said, for this project I keep hearing more and more info (e.g. 100 rows in the flat file and always in the same format) that is eroding away any "well what about..." questions that would make the 16 months make sense. Does the scope of the work include anything beyond parsing the data and "sending it for processing"? E.g. you mention that after the parse you must "compare with a set of fields from your own invoice in SQL"... is this just something like string matching (make sure field A from the file and value for field A in my SQL table are identical) or is there something more to it (if the address fields don't match between flat file and the invoice in SQL then use Google Maps API to find best well-formatted address including zip+4)? Is there any piece of the "processing" (you mentioned "send it for processing") that has to be built in this scope of work as well? You see where I'm going I'm sure. That being said, if the scope is as simple as you describe it and you dev is a real dev (not someone from ops who claims they program in excel or whatever) then I have a hard time believing this would take someone more than a week or two.

For $50, you could probably go to the fiver web site and have this done overnight from the other side of the world, ready to use when you get your coffee in the morning.

Forgot to mention 'fire the a$$hat who was supposed to be overseeing this clown.'

16 months is too long to cry BS. Barring any procedural holdups, it should have been pretty obvious after 30 days something was forked.

They just compare fields and if different send it back and if the same send it on.

Sounds like the person parsing the file to do the compare is doing it correct, just the IT manager of the project didn't know how or what tools they were using. The issue is receiving the file and determining if it is the correct hash and has the correct fields and the team doing that is unable to do so as of yet. They don't know why the code rejects some files and not others. They don't know how to send the response back if good and they don't know how to send the response back to let the company sending the file what is missing. They do know it does reject some files.

Just so you all know, this is still not live. They tried a couple weeks ago and forgot some stuff and lost 300,000 records. It didn't go well from there.

Well... that would depend on who forgot what stuff.... last time I "forgot some stuff in the code" the "stuff" I forgot wasn't listed in the specification/ticket for the project.

As a retired Senior Oracle DBA I had to do something similar several times, if I read you original post correctly. Someone wants to give you some data so they generate a flat or "ASCII text" file for you. What I did was to see if there were matching fields in the various tables to match the data items in the incoming data file (text). I would generally create a single "temp" file to put the data into then generate a fairly simple query to compare the data. If the new data was good then it was inserted into their tables. Some things to check:
1. Are the data types matching?
2. if there are "char" or "varchar" data types are they 2 bit or 4 bit each. Some foreign data items are 4 bit characters!

Hope it gets worked out as someone has made this much harder than it needs to be?
John

After testing, they forgot to turn on the part of the code that process the file before it gets deleted. Evidently they were not aware they needed to do this, but it is their code.

This isn't my project, but I know the PM on the production side, not the IT side.

Wildthing, FTP would be a better option than what they are (not) doing right now.

Honestly , I can't see people keeping their jobs after this. There is some background info I would love to give, but since this is a public thread I am hesitant to give some of that info out. Don't want to get the PM in trouble for me posting their internal struggles.

John, it isn't even as complicated as you are describing.

It sounds like one of the projects my wife was involved in at the university from which she retired.

One group was trying to move data to one department from the state government. When they finally showed my wife what they did, they were proud of it. Then she pointed out several violations of the law as to sensitive personal data and HIPPA violations. The entire project vanished, and several people were fired, and they started over from scratch. Government money just wasted.

Hell, I could do that in Bash

Crummy unit level testing in pre-release, that's on the dev team as a pre-release defect.

It's their data, but our code

Business owner - Recovery plan mode from the dev team NOW.

Oh yeah, I'm retired, everyday is a Saturday

FIFY

The plan from the devs was to do a manual fix for every record until they are up to speed. Sure, 3000k to 500k records per day. Should be no problem.