GSOC23 : Guidance for project "High-level HTTP client library"

Projects GSoC-2023
1

Hello Everyone,
I want to contribute to the project “High-level HTTP client library” for GSoC 2023, as mentioned in GSoC 2023 Contributor Instructions, we have to make a patch to a Fortran-Lang project, can anybody suggest me some repository and project related to above project idea, on which I should work.

Thank you

1 Like
2

Thanks, @rajkumardongre! For the purpose of applying to GSoC, you already have a PR going in the playground:

Which I will review and provide feedback over the next few days.

Otherwise, if you’re looking to contribute another PR, considering your interest you could review

and see if there is any (however small but meaningful) improvement you can make. Whether it’s a bug that you find and that you can fix, or add an example, or improve the docs.

4 Likes
3

Thanks @milancurcic for reviewing the code.till the I will explore the @interkosmos/fortran-curl repository and make contributions to it.

1 Like
4

I want to note that CURL is one of the most popular Internet client libraries (It does not only support http but among others, also email and ftp protocols) so there should be plenty of information (for other programming languages).

I think that it is quite important for Fortran that there exist bindings to the most popular libraries (or at least C libraries). Phillip (Interkosmos) really has put a impressive amount of work towards this goal for quite a large number of libraries.

6 Likes
5

And just in case it can be useful: Since Windows 10 curl and tar are also available in windows

3 Likes
6

Hello @community, I am interested in gaining more insight into “High-level HTTP client library” project .

As per the project description, the goal is to implement a high-level library for HTTP requests, similar to the Python requests library. This library aims to facilitate standard HTTP request methods (GET, HEAD, POST, PUT, DELETE, CONNECT, OPTIONS, TRACE, and PATCH) for Fortran programmers. So i think for this project we will create an fpm package, which will utilize fortran-curl, a set of Fortran bindings to libcurl, as a starting point. The package will include procedures for all standard HTTP request methods.

The intended outcome of this project is to enable easier consumption of HTTP web services from Fortran applications.

So end result will be something like this :
Let say in this project we have implemented a fortran package with name “http”.
In Code :

program http_demo
    use http                       ! we implement this http module
    implicit none
    type(response) :: res         
    character(20) :: url="https://httpbin.org/get", method="get"
    res = send_request(method,url, otherOptoins…..)
    print *, res%content          ! <html>….</html>
    print *, res%status_code      ! 200
    print *, res%ok		         ! .true.
end program http_demo

so here in this code we have use http module, create variable res which is instance of derived type response (it’s defination will be implemented in the http module itself), and call a procedure(or funciton) send_request, which has taken url and method as argument (it can take many argument like method url, params, data, headers, files, cookies, timeout and many more) and it will return a response type datatype, using which we are printing the response content, response status, using the property res%content, res%status_code and res%ok(it can have many property like content, cookies, headers, status_code, ok, url and many more).

So this is what I have understood about the project.

I would be grateful if you could provide me with further information, suggestions, and resources to better understand the project. Additionally, please let me know if I am going in the wrong direction with my understanding of the project.

Thank you for your time and consideration.

Sincerely,

Rajkumar

@interkosmos @milancurcic

2 Likes
7

Just some short remarks:

  1. It’s sensible to start with an API description before the actual implementation of such a library (Documentation-Driven Development), as the scope is not clear: what will the library solve? Which protocols should be covered? Only HTTP/S? How does the data model look like? Procedural or object-oriented API?
    Perhaps, you should investigate ways to describe the API first, and have a look at similar libraries in other programming languages, such as requests for Python. Consider the ways the response of a request could be consumed by the client (for instance, as a download to file system, or through a passed procedure interface for stream processing).
    Furthermore, some kind of abstraction around common HTTP request headers and content data (authentication, cookies, form data, …) would be handy.
  2. In your example, you’ve declared a response type only. How to we keep the request relatated information in case we want to re-send a request?
  3. How will errors be returned besides HTTP status codes?
  4. In case you aim for a procedural API, make sure to add a prefix to your procedure names, something like request.
  5. Names of derived types should contain a postfix, like _type or _t, i.e., type(response_type) instead of type(response).

Please let me know if you have further questions.

3 Likes
8

@interkosmos Thank you very much for your thoughtful and detailed remarks regarding the API description and implementation of the library. Your insights about Documentation-Driven Development, investigating ways to describe the API first, and considering similar libraries in other programming languages such as requests for Python are extremely helpful.

I will certainly take into account your suggestions and carefully consider them as I move forward with the development of the library. Once again, thank you very much for your valuable input, and please feel free to let me know if you have any further suggestions or comments.

9

Hello sir @interkosmos, I was reviewing the fortran-curl repository and noticed that there doesn’t appear to be a contribution guideline. Would it be possible to consider adding a contribution guideline to the README.md file? I believe that doing so could be helpful for new users who wish to contribute.

10

Hi, So I do some research on the HTTP library and reviewed similar libraries such as Python’s Requests module to gain a deeper understanding of the project. This was an amazing exercise that helped me understand how they implemented and abstracted the low-level code. As a result, I have created a brief API description for our HTTP module.

http_module API

The http_module is an object-oriented library written in Fortran that provides a simple interface for making HTTP requests to web servers. The library is built on top of the Fortran-CURL library, which provides a low-level interface to the CURL library for making HTTP/S requests.

To make our HTTP library fully abstracted and easy to use for all types of HTTP/S requests, we can create the following classes:

  1. HTTPRequest: This class will represent an HTTP request and encapsulate all the request details such as method, URL, headers, parameters, cookies, and request body.

  2. HTTPResponse: This class will represent an HTTP response and encapsulate all the response details such as status code, headers, cookies, and response body.

  3. HTTPClient: This class will be responsible for sending the HTTP requests and receiving the HTTP responses using the underlying HTTP client library (such as fortran-curl). It will take an HTTPRequest object as input and return an HTTPResponse object as output.

  4. string_dict: This class will be used to store key-value pairs where the key and value are both strings. It is similar to a dictionary or map data structure found in other programming languages.

By using these classes, we can create a highly abstracted HTTP library that provides a simple and consistent interface for sending and receiving HTTP requests and responses. This will make it easier for developers to use the library in their projects and handle different HTTP methods and response types.

HTTPRequest Class

Variables:

  • url : The URL to send the request to
  • method : The HTTP method to use (e.g. GET, POST, PUT, DELETE)
  • headers : A string_dict object containing any additional headers to send with the request
  • params : A string_dict object containing any query parameters to include in the request URL
  • data : The body of the request (e.g. for POST requests)
  • files : A string_dict object containing file data to include in the request
  • auth : The authentication details to use (e.g. username and password)
  • cookies : A string_dict object containing any cookies to include with the request
  • timeout : The time (in seconds) to wait for a response before timing out
  • verify : Whether to verify SSL certificates when making HTTPS requests

Procedure Interfaces:

  • set_method(method) : Sets the HTTP method to use for the request
  • set_url(url) : Sets the URL to send the request to
  • add_header(key, value) : Adds a new header to the request
  • set_headers(headers) : Sets the headers for the request using a string_dict object
  • add_param(key, value) : Adds a new query parameter to the request URL
  • set_params(params) : Sets the query parameters for the request using a string_dict object
  • set_data(data) : Sets the body of the request
  • add_file(key, value) : Adds a file to include in the request
  • set_files(files) : Sets the files to include in the request using a string_dict object
  • set_auth(auth) : Sets the authentication details to use for the request
  • add_cookie(key, value) : Adds a new cookie to include in the request
  • set_cookies(cookies) : Sets the cookies to include in the request using a string_dict object
  • set_timeout(timeout) : Sets the timeout for the request
  • set_verify(verify) : Sets whether to verify SSL certificates when making HTTPS requests
  • request : Take all the url, method, headers, cookies as argument and intilize the request object.

HTTPResponse Class

Variables

  • status_code: An integer representing the HTTP status code of the response.
  • headers: A string_dict object containing the response headers.
  • content: A string containing the response body.
  • cookies: A string_dict object containing the cookies returned in the response.

Procedure Interfaces

  • set_status_code(status_code): Sets the HTTP status code of the response.
  • get_status_code(): Returns the HTTP status code of the response.
  • set_header(name, value): Sets the value of the specified header in the headers dictionary.
  • get_header(name): Returns the value of the specified header from the headers dictionary.
  • set_content(content): Sets the response body to the specified content.
  • get_content(): Returns the response body.
  • set_cookie(name, value): Sets the value of the specified cookie in the cookies dictionary.
  • get_cookie(name): Returns the value of the specified cookie from the cookies dictionary.

HTTPClient Class

Variables:

  • curl_handle: CURL object handle to manage the HTTP request.
  • user_agent: User agent string to be sent in the request header.
  • timeout: Request timeout in seconds.
  • follow_redirects: Flag to follow redirects or not.
  • max_redirects: Maximum number of redirects to follow.
  • cookies: Dictionary to hold cookies to be sent in the request.
  • headers: Dictionary to hold headers to be sent in the request.
  • params: Dictionary to hold query parameters to be sent in the request.
  • data: Dictionary to hold data to be sent in the request body.
  • files: Dictionary to hold files to be sent in the request body.
  • response: Instance of HTTPResponse class to hold the response data.

Procedure Interfaces:

  • new(): Constructor to initialize the HTTPClient object.
  • set_user_agent(user_agent): Method to set the user agent string to be sent in the request header.
  • set_timeout(timeout): Method to set the request timeout in seconds.
  • set_follow_redirects(follow_redirects): Method to set the flag to follow redirects or not.
  • set_max_redirects(max_redirects): Method to set the maximum number of redirects to follow.
  • set_cookies(cookies): Method to set the cookies to be sent in the request.
  • set_headers(headers): Method to set the headers to be sent in the request.
  • set_params(params): Method to set the query parameters to be sent in the request.
  • set_data(data): Method to set the data to be sent in the request body.
  • set_files(files): Method to set the files to be sent in the request body.
  • get(url): Method to send a GET request to the specified URL.
  • post(url): Method to send a POST request to the specified URL.
  • put(url): Method to send a PUT request to the specified URL.
  • delete(url): Method to send a DELETE request to the specified URL.
  • head(url): Method to send a HEAD request to the specified URL.
  • options(url): Method to send an OPTIONS request to the specified URL.
  • get_response(): Method to get the response object for the last request sent.
  • send(request, response): Method sends a request to the server and store it in the response object.

Please let me know. If you have any suggestions or any other Strategy to implement the library.

4 Likes
11

Thanks for that, @rajkumardongre. I think it’s a great start. Many details would change during the potential project, but this is an appropriate level of detail for a GSoC proposal.

1 Like
12

Thank you @milancurcic

13

I was wondering if there exists a data structure in Fortran that resembles a MAP? Our project would greatly benefit from such a structure, particularly in setting headers, cookies, body, and other parameters. In case there isn’t any available, we would have to develop one ourselves

14

Yes, there are many libraries for that. Ones that come to mind are fhash by @lkedward and the hashmaps module in stdlib. To me fhash seems more approachable.

2 Likes
15

Excellent, problem solved

1 Like
16

Dear Fortran Community,

I have completed my GSoC 2023 proposal for the project “High-level HTTP client library”. The proposal aims to develop a high-level HTTP client library in Fortran that provides a simple and intuitive interface for performing HTTP requests.

I would like to take this opportunity to request your review of the proposal. Your feedback and suggestions are highly appreciated and will be instrumental in making the proposal more compelling and effective. If you have any suggestions or changes, please let me know, and I will make the necessary changes.

The proposal can be found Here. I would be grateful if you could take some time to review it and provide your feedback by 4th April.

I believe that this project has the potential to make a significant impact on the Fortran open-source community, and I am excited about the possibility of being a part of it. I am confident that, with your support and guidance, we can make this project a success.

Thank you for your time and consideration. I look forward to hearing from you soon.

6 Likes
17

Hi Everyone,
I am excited to share that I have received some fantastic news: my proposal for the “High-level HTTP client library” project has been accepted for Google Summer of Code (GSoC) 2023! This is a significant achievement, and I cannot wait to be a part of this thriving community and start working on the project. I am thrilled to have this opportunity to develop my skills and contribute to the Fortran community :star_struck:.

image
image771×585 25.2 KB

9 Likes
18

Note that there is also the libsoup 3 library (HTTP client/server library) which could be interfaced with Fortran:

With my cfwrapper Python script, I obtain a rather good result with 375 Fortran interfaces generated, and that could probably be improved a little more.

1 Like
19

Congrats @rajkumardongre! I’ll set up a poll to find a common weekly day and time for the project group (you and mentors) to meet. Stay tuned.

1 Like
20

Thank you, @vmagnin. I will take a look at it.

1 Like