a manic web-development framework
toolips is a software ecosystem that targets the full scope of web-development; from the most basic of APIs to complex full-stack applications. The goal of this project is to solve the two-language problem as it pertains to web-development — allowing all web-development to take place in one Scientific programming language. The head of this ecosystem is the Toolips web-development framework — a web-development framework designed with extensibility and flexibility in mind.
For example, Toolips offers fullstack callbacks through the ToolipsSession extension, enhanced SVG capabilities through the ToolipsSVG extension, and UDP servers through the ToolipsUDP extension. Julia, unlike JavaScript, does not easily run inside of a web-browser on the client side — so our Julia code is entirely server-side and servers like this are typically associated with back-end projects. Toolips is capable of far more than the average framework with extensions, and this includes full-stack development, so with this particular web-framework we are able to manage the front-end from the back-end. Alongside full-stack web-development, Toolips includes a plethora of alluring features:
HTTPS capable Can be deployed with SSL.
Extensible server platform.
Hyper-Dynamic Multiple-Dispatch Routing — The Toolips router can be completely reworked with extensions to offer completely new and exceedingly versatile functionality.
Declarative and composable — files, html, Javascript, and CSS templating syntax provided by ToolipsServables.
Modular servers — toolips applications are regular Julia Modules, making them easier to migrate and deploy.
Versatilility — toolips can be used for all use-cases, from full-stack web-development to simple endpoints.
Parallel Computing — Declarative process management provided by parametric processes.
Optionally Asynchronous — the Toolips.start! function provides several different modes to start the server in, including asynchronous, single-threaded, and multi-threaded.
Multi-Threaded — Toolips has support for high-level multi-threading through the ParametricProcesses Module
Toolips is able to create ...
Endpoints
File servers
Interactive fullstack web applications (using the ToolipSession extension)
Other HTTP or TCP servers (e.g. Proxy server, data-base cursor)
UDP servers and web-services services (e.g. Systems servers, DNS servers) (using the ToolipsUDP extension for UDP.)
To get started with Toolips from Julia, you will need to add the package to an environment with Pkg.
using Pkg
# best available build:
Pkg.add("Toolips")
# latest working:
Pkg.add(name = "Toolips", rev = "stable")
# latest dev:
Pkg.add(name = "Toolips", rev = "Unstable")
Toolips servers only require a Module to run, not a file-system — we can start a server from any Module with Toolips.start!. start! is provided with a Module (your server), an IP4 (the IP and port to start the server on,) and has the optional key-word arguments threads and router_threads.
module ExampleServer
using Toolips
# 'hello world' in Toolips:
main = route("/") do c::Connection
write!(c, "hello world!")
end
# exporting `main` will load the route. `start!` will allow us to start the server.
export main, start!
end
From here, we simply use the server and then start! it.
using Main.ExampleServer
start!(ExampleServer)
start! comes with a number of arguments to consider including the IP. The inverse to start! is kill!.
start!(MyToolipsApp)
kill!(MyToolipsApp)
An IP4 is constructed in a relatively pragmatic way... We provide a String (the address) and an Integer (the port) to :. This may be provided to start!
ip = "127.0.0.1":8000
start!(ExampleServer, "127.0.0.1":9000)
Though a file-system is not required, it may be easier to start from a base project and utilize a file-system or Pkg environment. For this, we may use Toolips.new_app, which takes a String, our project name. We can also provide a ServerTemplate to both start! and new_app in the first position before the other arguments to create or start a certain type of app. For example, the here is a simple ToolipsUDP server
using Toolips; Toolips.new_app("MyToolipsApp")
This will generate a simple base project, demonstrating the basics of Toolips:
module MyToolipsApp
using Toolips
# using Toolips.Components
# extensions
logger = Toolips.Logger()
main = route("/") do c::Toolips.AbstractConnection
if ~(:clients in c)
push!(c.data, :clients => 0)
end
c[:clients] *= 1
client_number = string(c[:clients])
log(logger, "served client " * client_number)
write!(c, "hello client #" * client_number)
end
# make sure to export!
export main, default_404, logger
end # - module MyToolipsApp <3
A dev.jl file is provided alongside these projects to automatically start a development server with a simple include. This server may be started with include("dev.jl"), or from Bash or Command Prompt julia -L dev.jl.
Note the export of default_404. default_404 is routed to 404 (try Toolips.default_404.path). to replace this response page simply make a new route with the 404 path and export it. default_404 will automatically be used on servers without a 404 route, so this export is reduntant but in place to demonstrate a 404 page.
—
Toolips features a dynamic three-stage routing system that can change functionality using parametric polymorphism. Two main functions comprise this routing system, route and route!. route is called to create routes, whereas route! is called by the server and routes the incoming Connection to a route. A standard Route is created using the route function, which will take a Function as the first positional argument and a route as the second.
home = route("/") do c::Connection
end
The provided Function will take one positional argument, an AbstractConnection. A standard Toolips route will hold the Connection type in a type parameter. The call above will return a Route{Connection}. With no annotation, this will automatically use any AbstractConnection. This is primarily used for Toolips multiple-dispatch routing.
A route will be passed a Connection whenever it is routed with route!. A Connection represents a client's entrance into the Function pipeline — each time a request is made to the server. The Connection stores the server's routes in its Connection.routes field and the server's data in its Connection.data field. The most vital function in association with the Connection is write!, which is used to write data to the incoming Connection as a response. Note that write! is not write, as this is a mutating write! — a write on a response cannot be reverted!
There are also several getter methods associated with the Connection, which may be used to retrieve data(click to reveal docstrings):
For storing cookies, the (HTTP) Cookie type will be used.
Here are some other functions for working with the Connection (click to reveal docstrings):
Many connections will also come with a convert! and convert binding. This is one of the ways we can implement a Connection extension.
The standard process for using the Connection involves crafting a response from a Route and then writing it to the Connection with write!.
module LandingPage
using Toolips
using Toolips.Components
home = route("/") do c::AbstractConnection
title_box = h2(text = "welcome to my page!")
top_header = div("top", align = "center", children = [title_box])
style!(top_header, "padding-top" => 25px, "height" => 25percent, "width" => 100percent,
"background-color" => "#1e1e1e")
content = div("main", align = "left")
style!(content, "padding" => 1percent, "width" => 100percent,
"background-color" => "#1e1e1e")
mainbod = body(children = [top_header, content])
write!(c, mainbod)
end
LOGGER = Toolips.Logger()
export LOGGER, home
end
Creating a response is a great place to start with Toolips. To create an app, we build a regular Module with Toolips as a dependency.
module SimpleServer
using Toolips
end
We create a route using the route function, which is provided a path (String) and a page (Function). The target will start at / and from there we may add additional paths.
module SimpleServer
using Toolips
home = route("/") do c::AbstractConnection
end
# make sure to export!
export home
end
We can write strings as a response with write!
write!(c)
... and we can also use ToolipsServables Component templating or other packages to produce our response.
main = route("/") do c::Toolips.AbstractConnection
main_box = div("main")
file_button_style = style("div.filebutton", "padding" => 5percent, "margin" => 2percent, "color" => "#1e1e1e", "font-weight" => "bold", "font-size" => 15pt,
"border-radius" => 5px, "border" => "2px solid #1e1e1e", "cursor" => "pointer", "transition" => 850ms)
file_button_style:"hover":["transform" => "scale(1.05)"]
style!(main_box, "position" => "absolute", "overflow-x" => "show", "width" => 40percent, "height" => 60percent, "padding" => 5percent,
"top" => 0percent, "left" => 0percent)
for md_filename in readdir(DOCUMENTS_URI)
presentable_name = replace(md_filename, ".md" => "")
comp_name = replace(presentable_name, " " => "-")
push!(main_box, div(comp_name, text = presentable_name, class = "filebutton"))
end
write!(c, file_button_style)
write!(c, body("mainbody", children = [main_box]))
end
Templating for Toolips is provided by ToolipsServables, aliased as Components in Toolips. ToolipsServables provides an API for templating CSS, HTML, and JavaScript from Julia, allowing us to create some pretty complex websites.
Every AbstractComponent (html element, css class or CSS animation,) will have a name. A Style will take a name style pairs:
my_style = style("div.sampleclass", "background-color" => "green", "color" => "white", "font-weight" => "bold", "font-size" => 13pt)
A regular Component will take key-word arguments and style pairs. Special amongst these are :children, :extras, and :text.
:text is the text of the element
:extras are extra components to be written before the element, as opposed to as a child of the element. This is useful for attaching scripts and other things to components.
and :children are the children of that element, which will be written inside of it. We add new children by using push!, or by providing them as an argument when we construct the Component.
A Component can also have its style modified using style!, and setindex!/getindex! work to retrieve any of these properties with a Symbol or String. There is also set_children!.
mycomp = h2(text = "hello!")
mycomp = Component{:h2}(text = "hello")
style!(mycomp, "font-size" => 12pt, "font-weight" => "bold", "color" => "orange")
mycomp[:text] = mycomp[:text] * " world".
new_container = div("sample-container", children = [mycomp])
set_children!(new_container, [mycomp])
push!(new_container, h3(text = "welcome to my julia website"))
There is a lot more this templating can do, but this overview will not go too into detail. For a more exhaustive overview of the features ToolipsServables offers, you can check out the documentation for the package here
Also note the use of measures (e.g pt) from ToolipsServables.
A File type is also provided by the Components Module, and like the Component this type may be written directly to the Connection. ToolipsServables also features the interpolate! function for interpolating files of different types. For serving files from routes, Toolips includes the mount function. mount will take a key-value pair, the HTTP target path we want to mount to and the file path to mount. This path can be file or directory; in the case of a directory, this call will return a Vector{Route} and in the case of a file it will return a single Route{Connection}. Both of these can be exported to be loaded into the server.
module QuickFileServer
using Toolips
fs = mount("/" => "shared_directory")
export fs
end
It is very straightforward, it might also be worth looking into interpolate! from Components, as well.
Multiroute is a Toolips feature that implements multiple dispatch routes for different types of incoming connections. We can create multiple handlers for different cases and the router will handle different incoming connections with the appropriate AbstractConnection. The running example for this is the MobileConnection.
module MobileFriendlyServer
using Toolips
using Toolips.Components
home_n = route("/") do c::AbstractConnection
write!(c, h1(text = "desktop user!"))
end
home_m = route("/") do c::MobileConnection
write!(c, h1(text = "mobile user!"))
end
home = route(home_n, home_m)
export home
end
But with Toolips extensibility, we could easily implement our own. For example, we could easily implement a POST handler using this multiple dispatch technique. For more information on writing Connection extensions, check out the connection extensions section.
Routing in Julia goes through a multi-step pipeline in which an extensible function, route!, is called three times.
route! is called on each server extension (<: of AbstractExtension) that has a route! binding.
route! is called on the Vector{<:AbstractRoute}, routing to the appropriate route.
Finally, route! is then called directly on a route from the previous binding.
Understanding this routing system allows us to easily change it. In order to change extension behavior each time the server routes, we use the first binding. In order to change the router, we would extend the second option and if we wanted to change what a Route does when routed we use the final option. For example, in order to make a hostname router we would use the second and third:
using Toolips
import Toolips: route!
abstract type AbstractHostRoute <: Toolips.AbstractRoute end
mutable struct HostRoute <: AbstractHostRoute
path::String
link::String
end
route!(c::AbstractConnection, r::AbstractHostRoute) = begin
proxy_pass!(c, r.link)
end
route!(c::AbstractConnection, vec::Vector{<:AbstractHostRoute}) = begin
host = get_host(c)
route!(c, vec[host])
end
main = HostRoute("sample.com", "127.0.0.1:8000")
export main
end
For more information on the first dispatch of route!, see creating server extensions
Another way to extend Toolips is by adding a Connection extension. This takes shape in a new AbstractConnection, which will need to have a binding to write! (or use the default binding,) and bindings to the Connection getter functions. For example, the IOConnection comes with Toolips and is used to translate client data from the header to the other threads in the ProcessManager, so it contains fields for all of this data:
mutable struct IOConnection <: AbstractIOConnection
stream::String
args::Dict{Symbol, String}
ip::String
post::String
route::String
method::String
data::Dict{Symbol, Any}
routes::Vector{<:AbstractRoute}
system::String
host::String
And then a binding for each getter
Note that if you replicate this behavior (storing the data as fields of your Connection,) making it a sub-type of AbstractIOConnection will automatically bind those getters.
Likewise, if our implementation is a bit simpler we could just use the http.Stream from the original Connection, as is the case for the MobileConnection.
mutable struct MobileConnection{T} <: AbstractConnection
stream::Any
data::Dict{Symbol, Any}
routes::Vector{AbstractRoute}
MobileConnection(stream::Any, data::Dict{Symbol, <:Any}, routes::Vector{<:AbstractRoute}) = begin
new{typeof(stream)}(stream, data, routes)
end
end
From here, if we could either implement this custom Connection into our Function pipeline, or write a new binding for convert and convert!. The MobileConnection does the former:
function convert(c::AbstractConnection, routes::Routes, into::Type{MobileConnection})
get_client_system(c)[2]::Bool
end
function convert!(c::AbstractConnection, routes::Routes, into::Type{MobileConnection})
MobileConnection(c.stream, c.data, routes)::MobileConnection{typeof(c.stream)}
end
With our convert! functions in place, we can create a multi-route with them for our server:
module MobileFriendlyServer
using Toolips
using Toolips.Components
home_n = route("/") do c::AbstractConnection
write!(c, h1(text = "desktop user!"))
end
home_m = route("/") do c::MobileConnection
write!(c, h1(text = "mobile user!"))
end
home = route(home_n, home_m)
export home
end
Then we may start the server:
start!(MobileFriendlyServer)
—
As touched on in our initial overview of start!, start! is the inverse of kill!. A call to start will return a ProcessManager. The ProcessManager is provided by ParametricProcesses and makes it easier for Toolips to distribute incoming tasks onto multiple threads. For more information of the process management system behind Toolips, it might be worthwhile to check out the ParametricProcesses documentation.
The start! function gives us several key-word arguments, async, threads, and router_threads will all change the thread mode of our server. async will determine whether or not the server runs asynchronously on the main thread. When set to false, an active server will take the entire thread. threads will determine the total number of workers in our ProcessManager. Finally, router_threads is used to determine the range of threads that should be used when routing the server. On a server with threads set to 1, router_threads will not be used. When router_threads is used, each number in the range below 2 represents execution on the base thread. To elaborate, in the case of -2:3 we would serve the first four incoming requests using the base thread and then use an additional two threads before repeating.
start!(mod::Module = Main, ip::IP4 = ip4_cli(Main.ARGS);
threads::Int64 = 1, router_threads::UnitRange{Int64} = -2:threads, router_type::Type{<:AbstractRoute} = AbstractRoute,
async::Bool = true)
Understanding the base thread and how it impacts performance is important. Evaluating everything on the base thread will always be faster, as we do not need to translate all of the data we are working with over to the other thread, unless that thread is already bogged down with a lot of other actions.
There are some key caveats to multi-threading in Toolips, at least when it comes to doing so using the router. A Stream can only be written to by the Base thread. Toolips provides a solution to this with the IOConnection. This allows for data to be translated back to the base thread and written asynchronously. This means that for multi-threading to work, we will need to annotate our routes with AbstractConnection, or some type of converted Connection type. We also need to use a full environment and project for multi-threading to work, otherwise the module and its environment cannot be translated to the other threads. Ensure the website is a regular Julia package, not an in-REPL Module — otherwise you will get an undefined error when a job is distributed.
If all of that is enough, it is also possible to create and distribute processes manually using the ParametricProcesses API through the Connection or directly. Toolips exports the essentials (click to view documentation, and note the Connection dispatches added by Toolips):
A new_job can be distributed to a ProcessManager or Connection directly using these functions.
using Toolips
# disable router multi-threading
pm = start!(Toolips, threads = 8, router_threads = 1:1)
job = new_job() do
@info "hello world!"
end
assign!(pm, 2, job)
ToolipsSession 0.4.5* also has a multi-threading system for callbacks specifically, and this might be better in some projects.
Server extensions are an integral part of Toolips that allows the server to load new advanced capabilities with a single export. In order to use a server extension, start by importing and constructing it — in this case, we will use the Logger — a ServerExtension that comes with Toolips.
module SampleServer
using Toolips
LOGGER = Toolips.Logger()
export LOGGER
end
After constructing, just like the routes we will want to export our new extension. Congratulations, the extension is loaded. Here is an example from the Toolips ecosystem using ToolipsSession:
module SampleServer
using Toolips
using Toolips.Components
using ToolipsSession
SESSION = ToolipsSession.Session()
LOGGER = Toolips.Logger()
home = route("/") do c::AbstractConnection
log(c, "hello world!")
mainbod = body("main")
# session callback using `ToolipsSession`
on(c, mainbod, "click") do cm::ComponentModifier
color_choice = randn(1:5)
selected_color = ("blue", "white", "orange", "red", "green")[color_choice]
style!(cm, mainbod, "background-color" => selected_color)
end
write!(c, mainbod)
end
export SESSION, LOGGER
export home
end
For quick access to on_start and route! functionality, we can also take advantage of the QuickExtension symbolic type. This allows for new extension types to be implemented without creating a new structure.
module SampleServer
using Toolips
import Toolips: route!, on_start
LOAD = Toolips.QuickExtension{:load}()
on_start(ext::QuickExtension{:load}, data::Dict{Symbol, Any}, routes::Vector{<:AbstractRoute}) = begin
@info "The server just started"
end
route!(c::AbstractConnection, qe::QuickExtension{:load}) = begin
@info "The server just served a client"
end
home = route(c -> write!(c, "hello!"), "/")
export LOAD, home
end
A ServerExtension able to change the server's functionality in a few key ways:
The extension can do something each time the server routes using a route! binding.
The extension can do something each time the server starts with an on_start binding.
And finally, an extension can hold data for the server within its fields.
Creating your own server extension is straightforward, first we make our ServerExtension type,
module ClientTrackServer
using Toolips
mutable struct ClientTracker <: Toolips.ServerExtension
clients::Vector{String}
ClientTracker() = new(Vector{String}())
end
home = route(Toolips.default_404.page, "/")
TRACKER = ClientTracker()
export TRACKER, home
end
Then we bind it to any Toolips functions we want to use (on_start, route!),
module ClientTrackServer
using Toolips
# be sure to import, or you will create `ClientTrackServer.on_start` instead
# of a new `Method`.
import Toolips: on_start, route!
mutable struct ClientTracker <: Toolips.ServerExtension
clients::Vector{String}
ClientTracker() = new(Vector{String}())
end
home = route(Toolips.default_404.page, "/")
TRACKER = ClientTracker()
# our new bindings:
function on_start(ext::ClientTracker, data::Dict{Symbol, Any}, routes::Vector{<:AbstractRoute})
# ensure `clients` is left empty (in case of restart).
ext.clients = Vector{String}()
end
function route!(c::AbstractConnection, ext::ClientTracker)
ip = get_ip(c)
# check if the client is currently counted:
if ~(ip in ext.clients)
@info "registering client: $ip"
push!(ext.clients, ip)
end
ip = nothing
end
client_count = route("/clients") do c::AbstractConnection
write!(c, length(TRACKER.clients))
end
export TRACKER, home, client_count
end
and we finish by exporting it in our server Module. Note that these bindings are not necessary but optional dependent on the desired functionality of the ServerExtension.
While Toolips primarily targets HTTP-based web-development, the package is not just a web-development framework, but also a server-development framework. For UDP servers, check out the ToolipsUDP extension. Toolips includes a built-in TCP server extension. To start from a TCP server template, use new_app(:TCP, name::String) dispatch of new_app.
using Toolips
Toolips.new_app(:TCP, "NewServer")
This will give us a nice minimal demonstration:
module NewServer
using Toolips
using Toolips: get_ip4, handler, read_all
main_handler = handler() do c::Toolips.SocketConnection
query = read_all(c)
write!(c, "hello client!")
end
# (try with `socket = connect(server IP4); write!(socket, "hello server!"); print(String(readavailable(socket)))`
export main_handler, start!
end
The Connection is then replaced with the SocketConnection, and the Route with the Handler. The Handler is an AbstractHandler, created using the handler function. TCP is a Connection-based protocol, and in most cases we would choose to keep this connection open. There is also get_ip4 for getting the connecting port alongside the IP address. The is_closed and is_connected bindings are useful for this.
main_handler = handler() do c::Toolips.SocketConnection
input_str = ""
while is_connected(c)
input_str = input_str * input_str
if ~(length(input_str) > 2 && input_str[end - 1:end] == "!\n")
continue
end
result = input_str[1:end - 3]
@warn result
if result == "password"
break
end
write!(c, "thanks for sending!")
input_str = ""
end
# second loop?
while is_connected(c)
break
end
end
There is also a convenience function provided for easily creating a handler, continue_connection.
