Wanted: open data suitable for a data science project!
Uncategorized
28
Posts
25
Posters
0
Views
-
Wanted: open data suitable for a data science project!
Every year, we ask our students 2nd bach to do a project. We give them a (big) dataset (preferably in JSON or a bunch of files combined), ask some research question, and ask them to perform data cleaning and exploration related to that question using #Rstats.
I've used most obvious choices, so I'm turning to Fedi to find new, interesting datasets. If you have no idea, sharing helps too!
@JorisMeys *mutters incoherently about Sankey diagrams*
-
Wanted: open data suitable for a data science project!
Every year, we ask our students 2nd bach to do a project. We give them a (big) dataset (preferably in JSON or a bunch of files combined), ask some research question, and ask them to perform data cleaning and exploration related to that question using #Rstats.
I've used most obvious choices, so I'm turning to Fedi to find new, interesting datasets. If you have no idea, sharing helps too!
@JorisMeys Have you offered them the Open Food Facts data already?
https://world.openfoodfacts.org/
https://prices.openfoodfacts.org/Contributions to the data quality are welcome: https://wiki.openfoodfacts.org/Data_quality
-
Wanted: open data suitable for a data science project!
Every year, we ask our students 2nd bach to do a project. We give them a (big) dataset (preferably in JSON or a bunch of files combined), ask some research question, and ask them to perform data cleaning and exploration related to that question using #Rstats.
I've used most obvious choices, so I'm turning to Fedi to find new, interesting datasets. If you have no idea, sharing helps too!
@JorisMeys Here are some links I prepared earlier https://tomstafford.github.io/psy6422/module-project.html#finding-data
-
Wanted: open data suitable for a data science project!
Every year, we ask our students 2nd bach to do a project. We give them a (big) dataset (preferably in JSON or a bunch of files combined), ask some research question, and ask them to perform data cleaning and exploration related to that question using #Rstats.
I've used most obvious choices, so I'm turning to Fedi to find new, interesting datasets. If you have no idea, sharing helps too!
@JorisMeys I've been playing with the BTO bird survey data lately - 1994-2024, several million survey counts, interesting structure, one or two data cleaning challenges, possible research Q's over species distribution change, climate change effects, either over single species or groups. https://zenodo.org/records/18598453
-
@JorisMeys You could use some animal tracking data, e.g. from movebank. We also have drone images of seals if you are interested.
@GeertAarts @JorisMeys there's also the LiveMouseTracker datasets, very detailed movement and interaction data of up to 4 lab mice in SQLite format.
See here for the project's website https://micecraft.org/lmt/,
and here for an example dataset: https://micecraft.org/lmt/download/20180110_validation_4_ind_Experiment_6644_e.sqlite(This project is cool for many other reasons, not least the ikea-style build instructions or the fact that they repurpose Kinect cameras for scientific work as high-res 3D video cameras).
-
Wanted: open data suitable for a data science project!
Every year, we ask our students 2nd bach to do a project. We give them a (big) dataset (preferably in JSON or a bunch of files combined), ask some research question, and ask them to perform data cleaning and exploration related to that question using #Rstats.
I've used most obvious choices, so I'm turning to Fedi to find new, interesting datasets. If you have no idea, sharing helps too!
@JorisMeys In quantum field theory, one has certain regular graphs, each of which evaluates to a real number through some very complicated integral. The task is to predict that number through correlations with properties of the graph (number of cycles, cuts, etc.). For almost 2 million graphs, these numbers are freely available. See last paragraph of https://paulbalduf.com/research/statistics-periods/
-
Wanted: open data suitable for a data science project!
Every year, we ask our students 2nd bach to do a project. We give them a (big) dataset (preferably in JSON or a bunch of files combined), ask some research question, and ask them to perform data cleaning and exploration related to that question using #Rstats.
I've used most obvious choices, so I'm turning to Fedi to find new, interesting datasets. If you have no idea, sharing helps too!
UK DEFRA has some https://uk-air.defra.gov.uk/data/data-availability
https://sensor.community/en/ has a large Air quality dataset.
There is also an archive of Scottish Datasets.
https://opendata.scot/some of that may be useful.
-
Wanted: open data suitable for a data science project!
Every year, we ask our students 2nd bach to do a project. We give them a (big) dataset (preferably in JSON or a bunch of files combined), ask some research question, and ask them to perform data cleaning and exploration related to that question using #Rstats.
I've used most obvious choices, so I'm turning to Fedi to find new, interesting datasets. If you have no idea, sharing helps too!
UK Office for National Statistics have lots of data sets which might be relevant for you 🙂
-
Wanted: open data suitable for a data science project!
Every year, we ask our students 2nd bach to do a project. We give them a (big) dataset (preferably in JSON or a bunch of files combined), ask some research question, and ask them to perform data cleaning and exploration related to that question using #Rstats.
I've used most obvious choices, so I'm turning to Fedi to find new, interesting datasets. If you have no idea, sharing helps too!
-
Wanted: open data suitable for a data science project!
Every year, we ask our students 2nd bach to do a project. We give them a (big) dataset (preferably in JSON or a bunch of files combined), ask some research question, and ask them to perform data cleaning and exploration related to that question using #Rstats.
I've used most obvious choices, so I'm turning to Fedi to find new, interesting datasets. If you have no idea, sharing helps too!
@JorisMeys use #GBIF species occurrence point-data (@gbif ). Also spatial, and combinable: https://geodaten.bayern.de
-
@JorisMeys I've been playing with the BTO bird survey data lately - 1994-2024, several million survey counts, interesting structure, one or two data cleaning challenges, possible research Q's over species distribution change, climate change effects, either over single species or groups. https://zenodo.org/records/18598453
@JorisMeys This https://b-rowlingson.gitlab.io/n8workshop/Birds/birds.html is the sample analysis I did for a workshop - its very much done for teaching purposes rather than scientific ones, but there's probably a lot of science could come out of it.
-
Wanted: open data suitable for a data science project!
Every year, we ask our students 2nd bach to do a project. We give them a (big) dataset (preferably in JSON or a bunch of files combined), ask some research question, and ask them to perform data cleaning and exploration related to that question using #Rstats.
I've used most obvious choices, so I'm turning to Fedi to find new, interesting datasets. If you have no idea, sharing helps too!
@JorisMeys I always find interesting data at https://www.ipums.org/
-
Wanted: open data suitable for a data science project!
Every year, we ask our students 2nd bach to do a project. We give them a (big) dataset (preferably in JSON or a bunch of files combined), ask some research question, and ask them to perform data cleaning and exploration related to that question using #Rstats.
I've used most obvious choices, so I'm turning to Fedi to find new, interesting datasets. If you have no idea, sharing helps too!
@JorisMeys Oh, Pirate Bay is full of training data. If it's good enough (and legal enough) for Meta it should be good enough for the rest of us.
😄
-
Wanted: open data suitable for a data science project!
Every year, we ask our students 2nd bach to do a project. We give them a (big) dataset (preferably in JSON or a bunch of files combined), ask some research question, and ask them to perform data cleaning and exploration related to that question using #Rstats.
I've used most obvious choices, so I'm turning to Fedi to find new, interesting datasets. If you have no idea, sharing helps too!
@JorisMeys Not sure if that's obvious enough but ECMWF offers a number of weather-related datasets. https://www.ecmwf.int/en/forecasts/datasets
-
Wanted: open data suitable for a data science project!
Every year, we ask our students 2nd bach to do a project. We give them a (big) dataset (preferably in JSON or a bunch of files combined), ask some research question, and ask them to perform data cleaning and exploration related to that question using #Rstats.
I've used most obvious choices, so I'm turning to Fedi to find new, interesting datasets. If you have no idea, sharing helps too!
@JorisMeys have you heard of Jeffrey Epstein?
-
Wanted: open data suitable for a data science project!
Every year, we ask our students 2nd bach to do a project. We give them a (big) dataset (preferably in JSON or a bunch of files combined), ask some research question, and ask them to perform data cleaning and exploration related to that question using #Rstats.
I've used most obvious choices, so I'm turning to Fedi to find new, interesting datasets. If you have no idea, sharing helps too!
@JorisMeys the VAST challenge would be perfect for this! https://github.com/vast-challenge
Background: https://datastori.es/data-stories-24-vast-challenge/
-
Wanted: open data suitable for a data science project!
Every year, we ask our students 2nd bach to do a project. We give them a (big) dataset (preferably in JSON or a bunch of files combined), ask some research question, and ask them to perform data cleaning and exploration related to that question using #Rstats.
I've used most obvious choices, so I'm turning to Fedi to find new, interesting datasets. If you have no idea, sharing helps too!
@JorisMeys It might be an obvious choice, but the UK Data Service hosts many datasets. A minority of the datasets are open, and they tend to be pretty clean already, but the massive scale of some of them makes them interesting.
-
@JorisMeys It might be an obvious choice, but the UK Data Service hosts many datasets. A minority of the datasets are open, and they tend to be pretty clean already, but the massive scale of some of them makes them interesting.
@JorisMeys For example, the 2019 World Risk Poll is a global survey of fears and attitudes towards risk. It has lots of demographic detail too so you can simulate the effect of different sampling strategies.
It's a 70 MB plaintext spreadsheet with 150,000 rows. It's large enough that in my last visit to the Apple Store, I could compare devices by manually timing how long it took to open the file. (It took 7 seconds on anything with an M4 chip; 5 seconds with an M5.)
https://datacatalogue.ukdataservice.ac.uk/studies/study/8739#details
-
@adenoz Thanks, I didn't know that one. Very interesting, and indeed the kind of data structure I am looking for. It's a bit far from their major (it's students Bio-engineering), so I might opt for another dataset closer to that if I find one. But this one is definitely flagged and stored for future use.
@JorisMeys Students in bio-engineering might enjoy finding out facts about macromolecular structures from the PDB.
Both the RCSB PDB and PDBe portals offer APIs to query the meta-data about structures deposited in the PDB:
https://www.rcsb.org/docs/programmatic-access/web-apis-overview
https://www.ebi.ac.uk/pdbe/pdbe-rest-api
The very basics let you replicate the online dashboards on these portals, showing number of entries deposited or released per year. But the APIs give access to every piece of meta-data in there, so you can really ask sophisticated questions. I played a bit with it a while ago, see some examples here: https://guillawme.github.io/insights-from-the-pdb/
-
Wanted: open data suitable for a data science project!
Every year, we ask our students 2nd bach to do a project. We give them a (big) dataset (preferably in JSON or a bunch of files combined), ask some research question, and ask them to perform data cleaning and exploration related to that question using #Rstats.
I've used most obvious choices, so I'm turning to Fedi to find new, interesting datasets. If you have no idea, sharing helps too!
@JorisMeys @genenetwork How big is big? For biology/genetics the GeneNetwork repository (https://genenetwork.org) has a lot of whole-transcriptome datasets. Each is not huge but some interesting analyses could also be done by combining them in creative ways. I’ve used this resource a lot for bioinformatics training.
Most of the datasets can be downloaded as flatfiles and the API returns JSON.
Gli ultimi otto messaggi ricevuti dalla Federazione
-
@Gina
At last, someone following the lead of sir Berners-Lee and his Solid Pods!
👏🏻👏🏻👏🏻
@activitypods
-
gpg --quiet --decrypt --no-tty --for-your-eyes-only
-
@fucinafibonacci @filippodb @spettacoli imho Pulp Fiction grandissimo film sul cinema di quel tipo lì, quasi un manifesto, tipico di quel pazzo di Quentin.
-
@cwebber What's telling, I think, is that all these people go on about how much they're doing and how great AI is to help them build more *but there's no actual demonstrable stuff being done.* I mean, if AI was some kind of Nx multiplier you'd think we'd be getting N times more actual functionality out of software but mostly it seems like the N multiplier only applies to blog posts about how AI multiplies their programming.
-
Steve Klabnik also had an interview on lobste.rs. There's a lot in it! It's a cool read! https://alexalejandre.com/programming/steve-klabnik-interview/
And then it gets to the AI part and he's just like "oh I don't write code anymore".
And notably Steve Klabnik has a lot to say about code, but it's *all in the past*.
Lots of brilliant people are becoming non-practitioners.
-
Love that Betazoids are always trying to get freaky in the weirdest ways.
-
-
@stevendb perhaps the worst year, so far. But, I can still have a little fun.
