Developers who venture into crafting their own async/await implementations in Rust may encounter the intricate task of implementing the Future trait for their custom types. Rust’s approach to async/await is nuanced, offering a stark contrast to languages like Go, which employ preemptive scheduling. Instead, Rust embraces lazy evaluation and cooperative scheduling, allowing developers to meticulously control the yield points to the executor. This level of control, however, introduces complexity in implementing the Future trait for custom types. The intricacies arise because .await can’t be invoked within a non-async function, necessitating the development of a state machine (or similar) for these custom types. This endeavor can be laborious and fraught with potential errors, difficult to maintain, and may prompt developers to opt for BoxFuture<T>, a choice that could compromise performance.

A: Sampling the program at specific times, and do some statistics analysis.

It can be one of the following:

• Reading Backtraces of the program for every 1000th cycles, and represent in flamegraph.
• Reading Backtraces of the program for every 1000th cache miss, and represent in flamegraph.
• Reading Backtraces of the program for every 10th memory allocation, and represent in flamegraph.
• Get return address of the program for every 10th memory allocation, and show counts for every line.

How to trigger a sample?

Kind of triggers:

I’ve recently been working on rewriting our small service with Rust for fun. One reason is to see if Rust, as a system programming language, is ready for cloud development in 2023. Another reason is that I wonder how much better it is compared to Python or Java in major cloud and data computing.

💡 There is a lot of discussion over which language to use as a serverless service. In my point of view, dynamic languages like Python lack compile-time checks, which can cause more runtime errors than static languages like Rust or Java.
Another reason is that Python and Java need a runtime process (JVM, CPython) to run your actual code, which means they can’t run natively like compiled Rust does. However, I’m not sure about the performance gap. Furthermore, Rust doesn’t require garbage collection, so ideally, a program’s heap size should fluctuate less than a garbage-collected language when running. That means Lambda or Function underlying infrastructure controller should be able to handle invocation or scaling more easily.
In terms of language paradigms, in my opinion, compared to traditional OOP, the modern ML (meta-language) family has a better design for scalable cloud services.

]]>https://300watts.me/posts/make-kafka-schema-easier/Wed, 26 Jan 2022 10:04:55 +0800Morrishttps://300watts.me/posts/make-kafka-schema-easier/TL;DR: You can find the script on my GitHub repository: kschema-table

Coordinating schema with Data Scientist

As a data engineer, you’ll need to collaborate closely with data scientists/domain experts to design data schemas. The optimal schema will depend heavily on the business domain and how the product is used. For instance, in a cybersecurity context, threat experts and data scientists would likely be the ones designing the data schema. They may work with infrastructure and data teams to define these schemas in a common format like, for example: Avro IDL (Avro Interface Description Language).

]]>https://300watts.me/posts/ssh-tunneling-summary/Thu, 10 Dec 2020 10:04:55 +0800Morrishttps://300watts.me/posts/ssh-tunneling-summary/There are three types of SSH port forwarding:

1. Local Port Forwarding

   Forwards a connection from the client host to the SSH server host and then to the destination host port.

2. Remote Port Forwarding

   Forwards a port from the server host to the client host and then to the destination host port.

3. Dynamic Port Forwarding

   Creates a SOCKS proxy server that allows communication across a range of ports.

Roles

• Client: Any machine where you can ssh to enable Port Forwarding
• SSH Server: A machine that can be SSHed into by Client
• Target Server: A machine to which you want to establish a connection, usually to open services on this machine to the outside world.

Notice: both Client and SSH Server can be Target Server, it doesn’t really need three machines to perform Port Forwarding! However, Dynamic Port Forwarding will not be only one Target Server, but it can be dynamically determined.

]]>https://300watts.me/posts/practicing-bisect-in-go/Tue, 05 May 2020 11:46:56 +0800Morrishttps://300watts.me/posts/practicing-bisect-in-go/In Python, we are used to using bisect to search for elements in the List. In Go, sort.Search() can also provide us with the same purpose. As in Python, the target array is sorted before searching.

go

arr := []int{9, 2, 1, 3, 6, 4, 5, 5}
sort. Ints(arr)

After sorting, we use sort.Search(), the first parameter is the search target, and the second is the function of the search condition. Here we input slice: [1 2 3 4 5 5 6 9], find the 5 closest to the left, and the program returns the index 4.

]]>https://300watts.me/posts/how-to-mail-the-movie-list-from-the-remote-host/Fri, 10 Jan 2020 17:04:55 +0800Morrishttps://300watts.me/posts/how-to-mail-the-movie-list-from-the-remote-host/Sometimes I upload videos for my family but don’t know if they already exist on the computer. At this time, I can think of a way to automatically email the titles of the videos already in the video library to myself for reference on a regular basis.

zsh

~/Downloads
➤ cat 2021-12-03-movie_list.txt -r :10
───────┬────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: 2021-12-03-movie_list.txt
───────┼────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ 2021-12-03 Movie List
   2   │
   3   │ CreationTime             Name
   4   │ ------------             ----
   5   │ 2021/11/13 下午 05:03:45 紅色通緝令
   6   │ 2021/11/12 下午 04:03:19 高玩殺手
   7   │ 2021/11/12 下午 04:03:17 007生死交戰
   8   │ 2021/11/12 下午 03:57:13 芬奇的旅程
   9   │ 2021/11/12 下午 03:46:12 沙丘
  10   │ 2021/9/21 下午 03:28:33  最佳销售员
───────┴────────────────────────────────────────────────────────────────────────────────────────────────

Gmail low security settings must be turned on

1. Go to Account Settings
2. Click Security in the list on the left.
3. At the bottom of the low security restriction click Turn on access.

In this way, emails can be sent through the smtp.gmail.com api.

]]>https://300watts.me/posts/common-python-and-c-11-usage-comparison/Sun, 12 May 2019 11:46:56 +0800Morrishttps://300watts.me/posts/common-python-and-c-11-usage-comparison/My Gist

Define Variables

• Python

python

a = 1
b = 0.0
c = 'xxx'

• C++

cpp

int a = 1;
auto a = 1;
double b = 0.0;
auto b = 0.0;
const char c[] = "xxx"; // C string (char array)
const char* c = "xxx";  // C string (pointer)
std::string c = "xxx";
std::string c("xxx");
std::string c{"xxx"};

Variable Scopes

• Python

python

global_var1 = 1
def func(arg):
    local_var = 2
    global global_var2
    global_var2 = 'xxx' # global
if arg:
    local_var2 = 0.5 # scope: function
    ...
    # if arg evaluates to True,
    # local_var2 is still accessible here.
    return False

• C++

cpp

int global_var1 = 1; // global
static std::string global_var2; // global in the current file
bool func(bool arg){
int local_var = 2;
global_var2 = "xxx";
if(arg){
double local_var2 = 0.5; // scope: if block
....}
// local_var2 is undefined here}

Reference

• Python

python

s1 = {"key1" : 100}
s2 = s1 # reference the same object
s2["key2"] = 200
print(s1["key1"])
# {"key1":100, "key2": 200}
s2 = {} # s1 is NOT changed, s1 and s2

# reference different objects now

• C++

cpp

std::unordered_map<std::string, int> s1 = {"key1" : 100};
std::unordered_map<std::string, int> s2 = s1; // copy the whole object
std::unordered_map<std::string, int>& s2 = s1; // reference the same object
s2 = std::unordered_map<std::string, int>(); // s1 is changed, s1 and s2 still reference the same object
// C++ 11
auto s2 = s1;  // copy the whole object (slow)
auto& s2 = s1; // reference the same object
// Now s2 is a reference to s1
auto s3 = s2;  // copy the whole s1 object. s3 is NOT a reference
const auto& s3 = s2; // reference the same s1 object

Conditional

• Python

python

if a == 1 and b == 2:
    pass
elif c == 3 or d == 4:
    pass
else:
    pass
if a == 1:
...
elif a == 2:
...
elif a == 3:
...
else:
...

• C++

cpp

if (a == 1 && b == 2){}
else if(c == 3 || d == 4){}
else {}
switch(a){
case 1:
    ...
    break;    // without break, will run case 2 as well
case 2: { // create a new scope if we need to define new variables
    int a = 100;
    break;}
default:  // it's good practice to always add this
...};

Namespace & Imports

• Python

python

# Import: File: app/bidders/ai_util.py
import app.bidders.ai_util
# Namespace: defined by directory structure
# Fully qualified names:
app.bidders.ai_util.func("xxx")

• C++

cpp

// Import: Files: app/bidders/ai_util.hpp & ai_util.cpp
#include "app/bidders/ai_utils.hpp"
// Namespace: not related to directory structure
namespace app {
    namespace bidder {
        namespace ai_util {
            void func(const char* str);
        }
    }
}
// Fully qualified names:
app::bidders::ai_util::func("xxx");

Loops

• Python

python

for i in range(100):
    pass
while cond:
...
a = [0, 1, 100]
for item in a:
    pass
b = {"key": 0.5, "key2", 1.0}
for key, val in b.iteritems():
    pass

• C++

cpp

for(int i = 0; i < 100; ++i){...}
while(cond){}
std::vector<int> a = {0, 1, 100};
for(auto& item: a){ // without &, this will copy each item
}
std::unordered_map<std::string, double> b = {{"key", 0.5}, {"key2",
1.0}};
for(auto& item: b){ // without &, this will copy each item
auto& key = item.first;
auto& val = item.second;}

Functions

• Python

python

def func(arg1, arg2):
    ....
    return ret1, ret2, ret3

• C++

cpp

void func(const Arg1& arg1, const Arg2& arg2,
Ret1& ret1, Ret2& ret2, Ret3& ret3
){...
ret1 = ....;
ret2 = ....;
ret3 = ....;}
No multiple return values
Need to specify the type of the return value
Every variable needs to have type declaration
Declaration before use is required
Add const to the references that are not changed by the method
Declare in *.hpp, implement in *.cpp (for public functions)

Class Definition

• Python

python

# Python version: only one *.py file:
class PythonClass(ParentClass):
    def __init__(self): # constructor, pass instance as self
        ParentClass.__init__(self) # python2
        self.attrib = 1111
        Self.attrib2 = 'xxx'
    def some_method(self, arg1, arg2):
        return arg1 * arg2 + self.attrib
    def _some_private_method(self): # kind of private func
        pass

• C++

cpp

// --------- Declaration: cpp_class.hpp ---------
class CppClass: public ParentClass {
public:
CppClass(): ParentClass(), attrib(5566),
attrib2("xxx"){}
virtual ~CppClass(): {
// destructor: free allocated resources here}
double someMethod(double arg1, double arg2);
private:
void somePrivateMethod(){}
int attrib;
std::string attrib2;};
// --------- Implementation: cpp_class.cpp ---------
#include "cpp_class.hpp"
double CppClass::someMethod(double arg1, double arg2){
return arg1 * arg2 + attrib;}

Virtual function

• Python

python

class Raccoon:
    def get_name(self):
        return "raccoon"
class Zebra(Raccoon):
    def get_name(self):
        return "zebra_" + Raccoon.get_name()
def func(maybe_racoon):
    print(maybe_racoon.get_name())
obj = Zebra()
func(obj)

# pintout: zebra_raccoon

• C++

cpp

class Raccoon {
public:
std::string getName() const {
return "raccoon";}};
class Zebra: public Raccoon {
public:
std::string getName() const {
return "zebra" + Raccoon::getName();}};
void func(const Raccoon& maybeRaccoon){
std::cout << maybeRaccoon.getName() << std::endl;}
Zebra obj;
func(obj);
// pintout: raccoon

• C++

cpp

class Raccoon {
public:
virtual std::string getName() const {
return "raccoon";}};
class Zebra: public Raccoon {
public:
std::string getName() override const {
return "zebra" + Raccoon::getName();}};
void func(const Raccoon& maybeRaccoon){
std::cout << maybeRaccoon.getName() << std::endl;}
Zebra obj;
func(obj);
// pintout: zebra_raccoon

Manage Objects

• Python

python

obj = ObjClass()
obj.method(arg)
obj.attribute = 100
obj2 = obj # reference the same object
# Manual delete is not needed

• C++

cpp

ObjClass* obj = nullptr; // prefer nullptr over NULL
ObjClass* obj = new ObjClass(); // allocate on heap
obj->method(arg);
obj->attribuge = 100;
auto obj2 = obj;   // point to the same object
auto obj2 = *obj;  // copy!
auto& obj2 = *obj; // reference the same object
delete obj;        // when not used, manual delete is required
ObjectClass localObj(); // allocate on local stack
localObj.method(arg);
// Raw pointer is not recommended. Use smart pointers
#include <memory>
std::shared_ptr<ObjClass> obj;
auto obj = std::make_shared<ObjClass>();
obj->method(arg);
obj->attribute = 100; // manual delete is not needed
auto obj2 = obj;      // point to the same object (no * or &)

Common Data Types (from Python to C++)

• int:
  • int, long, unsigned int, unsigned long (size is architecture dependent)
  • std::int64_t, std::uint64_t, std::int16_t, … (#include , well-defined sizes)
• bool:bool
• float: double (64-bit), float(32-bit, bad performance & not recommended)
• str, bytes: std::string (#include )
• containers:
  • list: std::vector<> (#include )
  • dict: std::unordered_map<> (#include <unordered_map>)
  • set: std::unordered_set<> (#include <unordered_set>)
• None:
  • For float, can use NAN (#include ) and use std::isnan(number) to check if it’s NAN
  • For string, just use empty string and use str.empty() to check if it’s empty

Define Strings

• Python

python

s = "this is a string"
s2 = s # s2 and s reference the same object
len(s)
t = "prefix_" + s + "_suffix"
t = "prefix1" + "prefix2" + s
s = "has\0zero"
# len(s): 8

• C++

cpp

#include <string>
std::string s = "this is a string";
auto s2 = s; // copy s to s2 (new object)
auto& s2 = s; // s2 is a reference only
s.length();
auto t = "prefix_" + s + "_suffix"; // works but slower
std::string t = "prefix_"; t += s; t += "_suffix"; // good
auto t = "prefix1" + "prefix2" + s; // does not work
std::string s = "has\0zero"; // incorrect
s.length(): 3
std::string s("has\0zero", 8); // correct
Alternative (C++ 14):
using namespace std::string_literals;
auto z = "has\0zero"s; // add "s" suffix, z is std::string
auto z = "has\0zero"; // z is char* pointer

String Methods

• Python

python

t = "test str"
if t.find("sub_str") == -1:
    print("not found")
u = t[1:2] # get sub string
u = t[2:]  # get sub string til end
if not t:
    print("empty str")
v = t.lower()

• C++

cpp

#include <string>
std::string t = "test str";
if (t.find("sub_str") == std::string::npos)
std::cout << "not found\n";
if(t.empty())
std::cout << "empty str\n";
auto u = t.substr(1, 2);
auto u = t.substr(1);
#include <algorithm>
#include <cctype>
std::transform(t.begin(), t.end(), t.begin(), std::tolower);
// This does not work in unicode

List (dynamic array)

• Python

python

a = [1, 2, 3]
b = ["str1", "str2", "str3"]
c = ["xxx", {}, 100, 0.5] # cannot be done in C++
a.append(100)
a.insert(2, 10)
del a[1], a[0:2]
tmp = a[0:2]
tmp = a[2]
tmp2 = b[1] # reference the element

• C++

cpp

#include <vector>
std::vector<int> a = {1, 2, 3};
std::vector<std::string> b = {"str1", "str2", "str3"};
std::vector<????> c // cannot be done in C++
a.push_back(100);
a.insert(a.begin() + 2, 10);
a.erase(a.begin() + 1);
a.erase(a.begin(), a.begin() + 2);
std::vector<int> tmp{a.begin(), a.begin() + 2};
auto tmp = a[2];
auto tmp2 = b[1]; // copy the element!
auto& tmp2 = b[1]; // reference the element

Set

• Python

python

a = set()
a = {"1", "2", "3"}
b = [1, 2, 3]
c = set(b)
a.add("x")
a.remove("2")
if "4" in a:
    pass

• C++

cpp

#include <unordered_set>
std::unordered_set<std::string> a;
std::unordered_set<std::string> a = {"1", "2", "3"};
std::vector<int> b = {1, 2, 3};
std::unordered_set<int> c(b.begin(), b.end());
a.insert("x");
a.erase("2");
if (a.find("4") != a.end()){
...}

Dict

• Python

python

d = {"a": 1, "b": 2}
nested = {
"a": {"a1": 0.5},
"b": {"b1": 0.3, "b2": 0.4},}
free = {"a": 100, "b": "xxx", 50: None} # cannot do this in C++
d = defaultdict(lambda: "null"); # cannot easily do this in C++ 

• C++

cpp

#include <unordered_map>
std::unordered_map<std::string, int> d = {
{"a", 1}, {"b", 2}};
std::unordered_map<std::string,
std::unordered_map<std::string, int>> nested = {
{"a": {{"a1", 0.5}}},
{"b": {{"b1", 0.3}, {"b2", 0.4}},};

Common Dict Operations

• Python

python

d["new_key"] = 100
d["no such key"] # raise KeyError
del d["key"]; gc.collect()
if "key" in d:
    e = d["key"]
for key, val in d.iteritems():
    pass

• C++

cpp

d["new_key"] = 100;
d["no such key"] // create a new item for it
d.erase("key");
auto iter = d.find("key");
if(iter != d.end()){
 // without &, this will do copy
auto& e = iter->second;}
 // C++ 11 ranged for loop syntax
for(auto& item: d){ // without &, this will do copy
auto& key = item.first;
auto& val = item.second;
...}

]]>