-
Notifications
You must be signed in to change notification settings - Fork 4
1. Basic folder structure
The repository is structured in folders and each of these folders has a specific function. The main development branch is under the development branch.
The repository contains the following folders:
This folder contains files and scripts for creating the database. Even though it contains MySQL scripts that create a basic structure of the database, during the development some tables were subject to change, mainly adding some fields. This was usually done manually by altering tables. Therefore ESID-main.sql, which was the main database creation file is at the moment obsolete. SocialSpy.sql was not changed and the scripts that are supposed to follow social innovation mentions and activity on Twitter can be still used as they are. However, even though initially existed idea that social media data about social innovation would be useful and followed, this idea did not materialize much further than data collection.
This folder contains all crawlers that were used to initially obtain automatically data from larger original data sources and to crawl websites. This folder contains two subfolders.
The first folder, called DownloadedDatabaseTransformers, contains scripts for getting data from structured data sources. These data sources were previously downloaded in CSV, json or XML format and then these scripts were used to parse information in these files and store them in database. The name of the data source files are for are presented in the name of the files.
The second folder called again ESIDcrawlers stores crawlers that were developed for general purpose crawling (crawling of the websites) and for crawling of data sources that could not be downloaded (the data could be retrieved only by accessing websites). The main general crawling script is called IndividualCrawlRunner.py. This script takes websites from the database and for each of the websites runs a crawler IndividualSpider1 (can be found in spiders folder) with the deadline of 10 minutes to execute and download all necessary pages. IndividualSpider1 as a framework is using Scrapy. The crawler runs with a depth of 3. Similarly WaybackMachineCrawlerRunner.py tries the resolve and download pages stored in way-back machine (internet archive). Inside the spiders folder are additional data source specific spires for crawling specific data sources. Several GeneralWebsite spiders are obsolete and currently, IndividualSpider is used for the main web crawling jobs is used.
Helpers folder contains scripts that were used for helping file and data manipulation. These scripts include the ones for parsing brat files from annotations, building statistics out of these, as well as exporting and creating dataset. Here are some of the important files in this folder:
- ExportToKNOWMAK.py - creates CSV files with projects, actors and other necessary things that can be sent to AIT for import into KNOWMAK tool
- ApplyUserSuggestions.py - applies changes that were suggested by users manually. It goes through user_suggestions table and applies suggested changes in the rows that do not have applied flag set and sents the applied flag.
- CheckInactiveWebsites.py - script that goes through websites in the database and sets InactiveWebsite flag to 1 if it recieves request status message larger than 400 (page not found+internal server error and some more).
- CreateAnnFiles.py - very simple file that helps with creating .ann files for annotations in brat format for a folder containing text files that should be annotated in the annotation task.
- CreateAnnotationDataSet_new.py - script that was used to generate annotation dataset. It was selecting projects that had more than 500 and less than 30000 words in the text. Currently it uses probably wrong Mongo collection, so needs an update.
- CreateNegativeDatasetSI.py - generates negative dataset out of excluded projects (projects that have Exclude flag set to 1. This will be mainly from DSI data source).
- ExportDSI_SI_DriveData.py - this file was generated for topic modelling. It exports DSI and SI-Drive projects with topics from original data sources.
- ExportDescriptions.py - exports project descriptions from the database.
- GetAllProjects.py - creates a spreadsheet with all projects.
- LanugageHistogram.py - checks language of the crawled data and makes histogram based on languages of the text.
- MakeHistogram.py - makes csv file with stats based on number of words per project (so later can be made histogram in excel).
- BratDataProcessing folder contains files for creating and analysing the dataset in brat format and a couple of neural network models.
- There are other files making different kinds of statistics.
Text mining folder contains a number of subfolders and files for extracting information from text, classifying them and topic modelling. Inside the folder are a number of folders and some files. The files inside the main folder that are important are:
- ApplyModelAndStore.py - used to apply social innovation classifier to projects in the database and save classification back to the database.
- ProcessMongoData.py - this processes MongoData with Stanford tagger in attempt to extract locations and organizations. It does not save to database, but just to a file. It is an unfinished method, as organization detection needs more filtering.
- TranslateAndConsolidate_new.py - a script that was used to translate and merge text to one massive mongo entry. These entries could be easier than processed on mass. Some processing was done on these entries, however, later in the project it was dropped as these Mongo entries don't reflect the structure of the website and for some processing, it was necessary to process only About or Contact us page and then go to more widespread approach in cascading style.
In terms of folders, the following are important:
This folder contains different kinds of text classifiers, mainly for use of social innovation criteria classification. The main files are in Trainers folder inside this folder. There can be found classifiers for Naive Bayes, Random Forests, SVM and neural networks as well as files used for calculating inter annotator agreement. Merged_dataset_trainers is a folder that was used to train classifiers based on all data at the end and contains a number of neural network based classifiers (LSTM, CNN).
Perform classification folder contains some files for testing classification models, however, not saving it to the database. Models folder contains trained models for the criteria.
NER folder contains everything related to named entity recognition. In this case it contains only files with experiments for location detection and resolution. The main file is FindLocationStoreSQL.py that used Stanford NER to extract the most probable city and country based on the frequency of appearance in the text.
This folder contains everything related to summarizer. Initial files are the ones developed by Dimitar with models created by him. Nikola_summarizer folder contains files that are Nikola's attempt to redevelop and improve the method.
In this folder are files that are for topic modelling. The main one that was used is GATE_Topic_Modelling_Webpages.py which retrieves text for the project and sents it to the GATE interface in order to retrieve topics and keywords.
This folder contains portal for users of the database where they can view, edit and suggest new projects. Most of the functions are in app.py and webpages.py files. Javascript is mainly in static/semanticus.js and static/main.js files. And templates are named quite right according to its function. This folder is using Flask.
This is the first initial interface for adding projects manually.
This folder contains another Flask interface for adding keywords, hashtags and users that should be followed on twitter. I believe also tweets could be viewed in the interface and a script that collects the tweets can be run through the interface.