September 27, 2024

I created my own search engine

Claudio Ctin2 mins
I created my own search engine

I have created my own search engine and I’ll tell you how it works.

First of all, I created my own web crawler. It runs on Python and stores various meta data in a mysql database.


SchBenedikt
 /
web-crawler

A simple web crawler using Python that stores the metadata of each web page in a database.

web-crawler

A simple web crawler using Python that stores the metadata of each web page in a database.

View on GitHub

The search engine, which I created with Bootstrap, then retrieves the results from mySQL. The bottom right always shows how long the query took.


SchBenedikt
 /
search-engine

The matching search engine to my web crawler.

search-engine

The matching search engine to my web crawler.

Features

Like Projects (The most liked pages are displayed at the top)
Display of the search speed.



View on GitHub

Please note that no robots.txt files are currently taken into account, which is why you cannot simply crawl every page.

What do you think of the project?

Feel free to write it in the comments!

Please follow and like us:
fb-share-icon
Tweet
Pin Share

Stiri similare

Python vs Java: A Deep Dive into the Best Programming Language for You

Python vs Java: A Deep Dive into the Best Programming Language for You

Claudio Ctin

Hey everyone! How’s your week going? 😊 Whether you’re in the middle of a coding marathon, enjoying a well-deserved break, or just here to explore new tech ideas, we’re happy to have you. Today, we’re diving into a hot topic: Python vs. Java. 🚀 These two programming giants are often at the center of debates,…

RMAG news

Using TypeORM with TSX: A Smoother Development Experience

Claudio Ctin

Working with TypeScript in Node.js applications can sometimes be challenging, especially when integrating with tools like TypeORM. While ts-node has been a popular choice for running TypeScript directly, it often comes with configuration headaches and performance bottlenecks. In this article, we’ll explore using tsx as an alternative to ts-node for TypeORM, providing a smoother and…

RMAG news

Next Js Localisation

Claudio Ctin

Hello All, I am new to this community and thank you for your support and time. I am working on a bilingual site (English and Japanese). I am using next-intl package to set the locale. I am trying to implement the following use cases Set the default language to user browser language Set the default…

RMAG news

Scale Deployments Based on HTTP Requests with Keda

Claudio Ctin

I’m using CloudWatch as a trigger, which means I have installed the CloudWatch agent responsible for sending metrics to CloudWatch. My KEDA fetches metrics based on the collection interval defined in the configuration of ScaledObject. You can follow the configuration below to set up the ScaledObject based on the number of HTTP requests. apiVersion: keda.sh/v1alpha1…

RMAG news

Human readable time in Rust

Claudio Ctin

When it comes to formatting time into the human-readable format in Rust, there could be various possible solutions including doing some math or using Instance. However, I demonstrate the millisecond crate, the dedicated and specialized crate for the purpose. This crate converts nanoseconds, microseconds, milliseconds, seconds, etc into short and long formats; suitable for human…

RMAG news

Introduction to the MERN Stack: Building Full-Stack Applications with Ease 🚀

Claudio Ctin

In the world of web development, the MERN stack has emerged as a powerful solution for creating dynamic and interactive applications. Comprising four key technologies—MongoDB, Express.js, React, and Node.js—this stack allows developers to build full-stack applications using JavaScript throughout. In this blog post, we’ll explore how these technologies work together, their pros and cons, and…

RMAG news

NEXT-JS PROJECT

Claudio Ctin

REVIEW In the course of two weeks I have gained skills needed to build a full-stack Next.js applications.On this project we were expected to build a full-stack web application which should have a public home page, login page, dashboard pages and setting up database. REQUIREMENTS . Operating system; MacOS, Windows or Linus . Node.js 18.17.0…

RMAG news

Special Types of Interviews: How to Identify and Succeed in Them..

Claudio Ctin

When preparing for interviews, it’s essential to recognize that not all interviews are alike. Some interviewers ask questions that follow specific patterns, and failing to identify these can make the process more challenging. Understanding these distinct types of interviews can help you better prepare and succeed. Let’s dive into the three main types: 1) Behavioral…

RMAG news

Essential Tips for Aspiring Web Developers

Claudio Ctin

Web development is an ever-evolving field that offers endless opportunities for creativity and innovation. Whether you’re just starting or looking to refine your skills, these tips will help you confidently navigate the world of web development. 1. Practice Regularly: Consistent practice is key to mastering web development. Set aside time each day to code, experiment…

RMAG news

Enhancing Healthcare Outcomes with Oracle Cloud Infrastructure: A Path to Better Results

Claudio Ctin

In today’s rapidly changing healthcare environment Leveraging advanced technology is critical to improving patient outcomes and operational efficiency. Oracle Cloud Infrastructure (OCI) offers a comprehensive suite of solutions designed to transform healthcare delivery by Improved data management Strong security guarantees and advanced analytics This article dives into how OCI drives better healthcare outcomes and why…

RMAG news

Mastering React: A Comprehensive Learning Guide

Claudio Ctin

Welcome to a thorough journey through React and its ecosystem! This guide will provide you with a structured path for mastering React, from understanding basic concepts to exploring advanced topics in React and associated technologies. Getting Started with React React is a powerful JavaScript library for building user interfaces, mainly for single-page applications where you…

RMAG news

IaC Security Analysis: Checkov vs. tfsec vs. Terrascan – A Comparative Evaluation

Claudio Ctin

Traditional, manual security processes can’t keep up with the speed of modern development, which leaves systems vulnerable to attacks. That’s where Security as Code (SaC) comes in. SaC automates security checks and policies, making them an integral part of the development pipeline. This ensures that security is built into every step without slowing down progress….

Terraform – Overview

Terraform – Overview

Claudio Ctin

What is Terraform? 🤔 Terraform is an open-source, cloud-agnostic, one of the most popular Infrastructure-as-code (IaC) tool developed by HashiCorp. It is used by DevOps teams to automate infrastructure tasks such as provisioning of your cloud resources. Terraform supported immutable infrastructure, a declarative language, a masterless and agentless architecture, and had a large community and…

RMAG news

Why is Nextjs so popular in startups?

Claudio Ctin

Nextjs is quite popular with startups. What are the main features of Next.js that make startups adopt it? Please follow and like us:

10 Essential Shadcn Components Every Developer Should Know About

10 Essential Shadcn Components Every Developer Should Know About

Claudio Ctin

Living in the web development world, it may just feel like digging for treasure until you find your tools. If you’re just starting now, well, you’re in luck because Shadcn is packed full of awesome components that can make life so much easier. If you are a professional or a beginner, knowing these 10 essential…

RMAG news

Execute Syntax on Background Process

Claudio Ctin

Pada postingan ini kita akan belajar cara remot ssh dan menjalankan perintah di balik layar atau background proses dengan 2 tools. Tools yg pertama akan kita bahas adalah tmux dan yg kedua adalah screen. Kedua alat tersebut adalah terminal multiplexer yang digunakan untuk multi tasking di terminal Linux. Okeh kita bahas cara penggunaanya. Saya asumsikan…

RMAG news

Scala is one of the best ways to learn Haskell

Claudio Ctin

First, a reminder: You DON’T NEED TO learn Haskell — that’s not what I want to talk about. I genuinely want to talk about learning Haskell, and Haskell specifically — not functional programming. The following might interest you if you want to get into Haskell for personal reasons (if you think it’s cool, well-paid, or…

What is the Difference Between Radix UI and ShadCN?

What is the Difference Between Radix UI and ShadCN?

Claudio Ctin

Be it a seasoned pro or a newcomer, UI library selection could make all the difference in any of your projects. Today, let’s take a deeper look at two of the top contenders: Radix UI and ShadCN. Both these libraries have something different to bring to the table, and by the end of this post,…

RMAG news

AuthorizationEndpoint vs TokenEndpoint

Claudio Ctin

In OAuth 2.0, the AuthorizationEndpoint and TokenEndpoint serve different roles in the process of obtaining access to resources on behalf of a user. Here’s a breakdown of the differences between them: 1. Authorization Endpoint Purpose: The AuthorizationEndpoint is responsible for obtaining authorization from the user to access their resources. This is where the user grants…

Custom Bootstrap 5 Breadcrumbs -Ver 2

Custom Bootstrap 5 Breadcrumbs -Ver 2

Claudio Ctin

Custom Breadcrumbs for Bootstrap 5 framework Abstract: We are presenting code (CSS) for custom Bootstrap 5 breadcrumbs. This is an improved version of the previously published article. 1 The need for better Breadcrumbs Bootstrap 5 framework is coming with very basic Breadcrumbs implementation. I needed something much better, both visually and more functional. Over time,…

RMAG news

A Comprehensive Guide to Appium 2.0 Migration

Claudio Ctin

In mobile app development, keeping up with the latest advancements in testing tools is crucial. A significant upgrade in automation testing is the release of Appium 2.0. With its myriad enhancements and new features, migrating to Appium 2 is essential for teams aiming to maintain a competitive edge. This Appium 2.0 Migration blog will guide…

FluxCD – A lightweight GitOps CD tool: Day 44 of 50 days DevOps Tools Series

FluxCD – A lightweight GitOps CD tool: Day 44 of 50 days DevOps Tools Series

Claudio Ctin

Welcome to Day 44 of our “50 DevOps Tools in 50 Days” series! Today, we are exploring FluxCD, one of the most popular tools in the GitOps ecosystem. FluxCD automates the process of deploying applications to Kubernetes, allowing for seamless continuous delivery and progressive deployments. It uses Git as the single source of truth, ensuring…

🌐 SSL Certificates and How to Implement Them in Your Website 🔐

🌐 SSL Certificates and How to Implement Them in Your Website 🔐

Claudio Ctin

In today’s web development world, ensuring security is critical. As developers, safeguarding user data and building trust with our audience are essential. One powerful way to achieve this is by implementing an SSL (Secure Sockets Layer) certificate on our websites. 🔒 In this post, we’ll dive into: What SSL is 🤔 Why it’s important 🌟…

Spring Boot RestTemplate getForEntity method

Spring Boot RestTemplate getForEntity method

Claudio Ctin

Producer application Dependencies Spring Web and Lombok pom.xml <?xml version=”1.0″ encoding=”UTF-8″?> <project xmlns=”http://maven.apache.org/POM/4.0.0″ xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xsi:schemaLocation=”http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd”> <modelVersion>4.0.0</modelVersion> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>3.2.5</version> <relativePath/> <!– lookup parent from repository –> </parent> <groupId>com.example</groupId> <artifactId>ProducerApp</artifactId> <version>0.0.1-SNAPSHOT</version> <name>ProducerApp</name> <description>Demo project for Spring Boot</description> <properties> <java.version>17</java.version> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter</artifactId> </dependency>…

RMAG news

Mastering Terraform: How Loops and Conditionals Unlock Smarter Infrastructure Automation

Claudio Ctin

In the ever-growing world of cloud infrastructure, managing and automating deployments has become increasingly complex. This is where Terraform comes in — as one of the most powerful tools in Infrastructure as Code (IaC), Terraform allows engineers to declaratively define and provision infrastructure with ease. However, as configurations grow, keeping them concise and manageable is…

The Power of Machine Learning in Angular: A Developer’s Guide

The Power of Machine Learning in Angular: A Developer’s Guide

Claudio Ctin

The intersection of Machine Learning (ML) and Artificial Intelligence (AI) with Angular is a game-changer for modern web applications. This article delves into how you can integrate AI and ML into your Angular projects, breaking it down with practical examples, real-world use cases, and helpful tips to guide you along the way. please subscribe to…

RMAG news

Introducing HTPX: A Lightweight and Versatile HTTP Client for JavaScript and Node.js

Claudio Ctin

As developers, we often need a reliable and efficient HTTP client for our web applications, whether we’re building with JavaScript in the browser or Node.js on the server side. That’s why I created HTPX — a powerful, lightweight solution designed to simplify HTTP requests while offering a range of features for modern development. In this…

RMAG news

Advanced Terraform Module Usage: Versioning, Nesting, and Reuse Across Environments

Claudio Ctin

Terraform, HashiCorp’s popular Infrastructure as Code (IaC) tool, has transformed how infrastructure is provisioned and managed. One of its most powerful features is the ability to create and use modules — reusable configurations that encapsulate sets of resources. For teams looking to scale and manage their infrastructure efficiently, mastering advanced Terraform module usage — including…

10 sec for UX Design Principles.

10 sec for UX Design Principles.

Claudio Ctin

DFFH RULE 1.D- Doherty Threshold : Users somehow replicate the computer’s working speed to navigate the components in the design. Tend to keep the waiting time for users at less than 400 ms. The loading time can be utilized to show visual animations of loading or task-processing phenomena to build trust. 2.F- Fitt’s Law :…