Topic: EU Tenders | Elvis

[victor]

By the way, the updated standard forms (starting to be used right now, available at http://simap.ted.europa.eu/en/web/simap/standard-forms-for-public-procurement) don't repeat the same mistake as previous ones and have a repeatable company / buyer section so that each member of a consortium has one for himself. I.e. the splitting of consortia means changing from the "old" format to the "new" format.

I'm going to ping @georgiana-b here, she's doing the heavy lifting with this

Thanks for the input, it would be great to have you around on the hack day at least. I feel there's a lot to talk about on this topic, prototype methods and try to build solutions.

As for the methods we've employed so far in identifying entities, I'll leave it for G to answer, here or live in case you decide to join our happy hacking session.

[georgiana-b]

I'm very happy about the new forms. I've been thinking who to warn about the issues with the old ones for a while. I'm currently experimenting with some machine learning techniques to identify and split the consortiums and I hope to find a way to handle those on a long term.
Would it be possible to enforce validation of a company name based on the national registration number? For example auto complete the name (and potentially the rest of the info) from the Registry of Commerce database based on the registration number. One of the biggest problems we have now is that companies write their name differently in different contracts and it's hard to asses if it's the same company or not. I understand this depends more on the implementation than the standard but if there are specs for how the procuring platforms should be built or some authority responsible with reviewing them, it would be a huge improvement to add this functionality.

[jachym_hercher]

Hi everyone,

I will try to make it to Mechelen - you can't get much closer than that.

@georgiana-b, we try to explain the issues with the old ones in the documentation to the data on the EU open data portal, but there's a only a certain amount of information you can include without flooding people. Nevertheless, we're always happy for feedback on how to make the data more user-friendly.

I'm currently experimenting with some machine learning techniques to identify and split the consortiums and I hope to find a way to handle those on a long term.

Happy to learn more on this. We're running a fairly large project where we try to use machine learning to identify unique entities - addressing exactly the issue you mention - so we might have some synergies in our work. Regrettably, we can't really enforce the IDs, because some participants in public procurement in some countries simply don't have them, but we are planning a few things to increase their usage. However, it'll still take some time.

[ada]

The future of Elvis!

[victor]

URL: elvis.tenders.exposed

User: dataharvest@tenders.exposed

Pass: DataHarvest

[p3nny]

https://drive.google.com/file/d/0B9Y0dJ9d0xkBeVRCYVdUdzJ0U0E/view?usp=sharing

[georgiana-b]

Dataset of soil cleaning contracts in EU 2012-2016: https://drive.google.com/file/d/0B58kVdotYLhDTHcyV2paMk05bjA/view?usp=sharing

[ada]

Jachym,

we are looking into the data you have published, but it seems it's not complete.

A subset of Tenders Electronic Daily (TED) data covering public procurement for the European Economic Area, Switzerland, and the former Yugoslav Republic of Macedonia from 2006-01-01 to 2015-12-31 in comma separated value format.

@georgiana-b had a look and the subset for 2015 contains 103623 contract award notices. Our dataset contains 706555.
Is it possible to upload the whole dataset?

[jachym_hercher]

Hi Adriana, the "subset" is just about the number of available "fields" ("columns"). All the relevant notices should be included. (I'll clarify that in the description at some point, thanks for bringing it up.)

The numbers are weird. The number of contract notices for 2015 should be around ~175 000 (based on our queries as well as the search you get from the official http://ted.europa.eu/ website), which corresponds to neither of the numbers :). A few people at the hackathon had a problem with downloading the data, because the download got somehow interrupted and stopped before everything was downloaded. (But the .csv stays there, just smaller, which might be confusing.) I think that could be the issue - can you or @georgiana-b check whether you get the same file size of the .csv if you download it twice?

[georgiana-b]

Hello Jackym,

You are right, the first time it's downloading the buffering stops before completion but subsequent downloads all have 539587 rows for the 2015 contract award notices which seems like a more realistic number.
I will cross reference it with our dataset to get the true number which is probably somewhere in between.
Thank you for your help!

[victor]

check whether you get the same file size of the .csv if you download it twice?

Maybe you guys can also publish the file size (MB and/or number of rows) and optionally a checksum? It's not uncommon to have interrupted downloads on medium to large datasets, but it's a good practice to provide a way to verify the payload.

@jachym_hercher What kind of help would you need on these data sets now? We already know about entity extraction (always a topic), but is there anything else that we can participate in to speed up the process? I'm offering a slice of my time and my techie skills if there's anything doable there

[jachym_hercher]

@georgiana-b Np, good. Concerning the other discrepancy (~500k vs. ~175k), this is just (the extremely confusing) terminology. Each "contract award notice" has multiple "contract awards", so when the CAN file has 500k rows, these are actually 500k "contract awards", not "contract award notices". I think this point is not important for your network visualisation, because you work only with "contract awards", but if you're interested, it's explained in some detail in the open data documentation.

@victor Yes, I will definitely add the row number / MB size, hopefully next week with a new re-upload. (There is actually also a bunch of smaller mistakes that I need to correct as well, sorry for that. Nothing that should mess up your work though.) Concerning the checksum - what exactly does that mean for a .csv? I assume you don't mean that I should simply add a line into the csv counting the number of other lines, right? (Sorry if it's a very naive question :). )

Thanks for the offer, always appreciated! I'll think about it and come back to you :).

[victor]

Concerning the checksum - what exactly does that mean for a .csv? I assume you don't mean that I should simply add a line into the csv counting the number of other lines, right? (Sorry if it's a very naive question :). )

Don't let yourself sidetracked by this too much, since it's not of vital importance. Here's what I meant, sorry for being vague (quote from the Wikipedia page):

A checksum or hash sum is a small-size datum from a block of digital data for the purpose of detecting errors which may have been introduced during its transmission or storage. It is usually applied to an installation file after it is received from the download server. By themselves checksums are often used to verify data integrity, [...]

[jachym_hercher]

Yeah, I read that, but it still wasn't clear to me what it means to a .csv :). But anyway, I agree, the number of rows should be enough.

[jachym_hercher]

@jachym_hercher What kind of help would you need on these data sets now? We already know about entity extraction (always a topic), but is there anything else that we can participate in to speed up the process? I'm offering a slice of my time and my techie skills if there's anything doable there

@victor I've thought about it a bit and while there are also some other areas, truth be told, the entity extraction is probably the most useful topic right now :). Where are you with this?

[victor]

Where are you with this?

Experimenting a lot, trying to work together with people in OpenNames – we all know each other and would be nice to hit multiple targets with one common project.

If that won't work, we also have some experiments that gave satisfying results (machine learning strategies), but they all need trained models on our entity types. And that's a lot of repetitive work we're trying to either simplify or avoid completely, so far without any success. Perhaps one day we'll both start to tag loads of entities and train the algorithm properly, but that's a bit painful to do.

Is there any reliable source for all commercial entities registered in EU member countries? Or at least one for each, that we could have access to? So far that was the most painful thing, having no data set to compare to in order to normalize the names for extraction (and more).

[georgiana-b]

@jachym_hercher We are trying to figure out the correspondence between the headers of data.europa dataset and the openspending dataset that we currently use. We are having some difficulties identifying the value fields (Ex: VALUE_EURO_FIN_1_1). If you have some time, it would be very helpful if you edit this spreadsheet with a short description/part of the form/correspondent in OpenSpending dataset for those value fields.

[jachym_hercher]

Sorry, a small clarification from my side: by "entity extraction" I meant just the issue of splitting consortia / joint procurement into individual organisations. That's the part we've ignored so far and where I'd be most interested what you're up to. By the way, what language do you use ? (Our machine learning algorithm for "unique entity identification" is in Java.)

Is there any reliable source for all commercial entities registered in EU member countries? Or at least one for each, that we could have access to? So far that was the most painful thing, having no data set to compare to in order to normalize the names for extraction (and more).

Nothing amazing. Digiwhist is working on finding information from the different business registries, you have http://opencorporates.com/, but nothing of that is easy.

@georgiana-b Check the documentation :-). E.g. "VALUE_EURO_FIN_1_1" does not have a correspondence, because it is marked as [ADDED], i.e. a field not coming directly from the forms. I had a quick look in the spreadsheet and marked the fields which I know are coming from our own calculations, but I don't know the openspending dataset, so I can't (easily) help you out...

[ada]

Jachym. I can't find the following CAN in the dataset
http://ted.europa.eu/udl?uri=TED:NOTICE:405247-2013:TEXT:EN:HTML&tabId=0

I wonder why?
And are you going to add the number of rows / size of the file? It'd be quite helpful.

Thank you!

[georgiana-b]

@jachym_hercher Thanks a lot. Apologies, I missed the documentation somehow but now I noticed it was on display the whole time.
As for the consortia, I made some experiments using Python and Google's SyntaxNet which is a part-of-speech identifier that also uses search. It has been very accurate in identifying the proper nouns and I'm trying to built an algorithm based on syntax + comparison with other similar strings that calculates the probability of a phrase to be a consortium.
I also made some less successful experiments in the past using Java and StanfordNer which unfortunately doesn't work very well for us because the phrases are out of context and it was trained to work on proper sentences. I'm considering giving it another try after putting the phrases in a made up context (making a sentence out of the form + data) if the method above doesn't work.