The idea behind the app is that there is a truck load of news articles written every day. Some have real information while many are just clickbaity headlines with just fluff in them.
The first problem is what to actually read among these news articles. Instead of spending time finding articles to read, the app presents you curated articles popular on the web with a good amount of discussion already in place about the event.
The second problem is the content. Its not always desirable to read the entire article when there are only a couple of points of importance in the article. Except for long form articles, a couple of points from the article are sufficient to get a gist of the article. The app presents a series of key points from the article generated automatically using summarizing algorithms.
A Python script as a daemon on the server instance runs 24x7 aggregating popular news articles from the web. It summarizes the articles and adds it to a mongoDB collection. The script can be found here.
The server runs on Node.js with a process manager PM2. It responds to API calls with JSON data. This part is the backend and is completely decoupled from the application (frontend) that you are "seeing". So in essence it doesn't care where the data is being sent, it could be a website (like now) or a mobile app (added to TODO) or even a microwave (hey, you never know).
The server also has redis. redis is an in-memory database, which is basically database in RAM. We use this as a cache, as the data isn't that volatile. redis has data structures in database, so we create a hashmap and store our data in it using a combination of category and page number as key. We also run a script to flush this hashmap every hour else it will never be updated. The result is instead of 50 milliseconds to serve one request, the server takes 5 milliseconds at maximum. (not that the server is that busy, but why not?)
Next is AngularJS which is responsible for the single page application. Notice except this page, there are no redirects on the website. For each category and page number there are separate API calls returning JSON which is displayed on the template.
Salmaan is a graduate student at San Jose State University studying software engineering.
Find more about him and his projects at his website or using the following social media buttons*,
v1.0 - Jan 4th, 2017
v1.1 - Jan 5th, 2017
- Added changelog
- Can search through multiple pages.
- Theme persists on reload.
- Added routing.- Current routes are: /worldnews, /technology, /science, /india
v1.2 - Jan 10th, 2017
- Added unique pages for each article
- Added social media share links
- Changed button layout
v1.3 - May 31st, 2017
- SEO stuff.
- Created structured data layout according to google's layout. Doesn't seem to be working though.
- Made the website scalable. (yes! for all of you 10 active users 🤗 )
- Moved database to the cloud, added a loadbalancer to the mix and turned on autoscaling.
- Added new categories!- New routes are: United States, United Kingdom, Canada, Europe, China, Uplifting News Could use a lot more of the last one
- Add RSS Feeds
- Tweak UI
- Analyze data, use clustering algorithms for tagging.
- Implement better searching, I am thinking cosine similarity search, need to figure out how to set it up.